Image processing method

ABSTRACT

An image processing method adopted to remove noise present in an image, includes: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which a plurality of band-limited images with resolutions decreasing in sequence are generated by filtering the input original image; a first noise removal step in which virtual noise removal processing is executed individually for each of the band-limited images; a second noise removal step in which actual noise removal processing is executed for the individual band-limited images based upon the band-limited images from which noise has been virtually removed through the first noise removal step; and an image acquisition step in which a noise-free image of the original image is obtained based upon the individual band-limited images from which noise has been actually removed through the second noise removal step, and the virtual noise removal processing executed in the first noise removal step and the actual noise removal processing executed in the second noise removal step are differentiated in correspondence to a frequency band of a band-limited image.

This application is a continuation of International Application No.PCT/JP2007/057186 filed Mar. 30, 2007.

INCORPORATION BY REFERENCE

The disclosures of the following priority application and Internationalapplication are herein incorporated by reference: Japanese PatentApplication No. 2006-096986 filed Mar. 31, 2006; and InternationalApplication No. PCT/JP2007/057186 filed Mar. 30, 2007.

TECHNICAL FIELD

The present invention relates to an image processing method that may beadopted when removing noise from an image or emphasizing edges in theimage.

BACKGROUND ART

There are methods known in the related art adopted to acquire anoise-free image by transforming an original image to reversiblemultiple-resolution images and reconstructing the image throughadjustment of the transformation coefficients. A typical example of sucha method is disclosed in patent reference 1. Patent reference 1discloses a method for acquiring a noise-free image by spatiallyfiltering for noise removal, the high-frequency subband coefficients inthe multiple-resolution images resulting from the transformation andthen inverse-transforming the image. Patent reference 2 discloses acontrasting method through which noise is sequentially removed fromreduced images on the low-frequency subband side, which are generated ona temporary basis during the process of the multiple resolutiontransformation.

Patent reference 3 discloses a method through which, instead of removingnoise directly on the transformation coefficient planes of themultiple-resolution images resulting from the transformation asdescribed above, noise is removed by extracting noise componentscontained in the high-frequency subband transformation coefficientscorresponding to LH, HL and HH resulting from the multiple resolutiontransformation via orthogonal wavelets, combining only the noisecomponents through inverse wavelet transformation and subtracting thesynthesized noise component from the original image, so as to assurebetter ease in the handling of the noise components. This method allowsthe extent of noise removal to be adjusted easily by applying a constantmultiple factor dependent upon the image structure in the real space tothe synthesized noise component prior to the final subtractionprocessing.

Patent reference 3 also presents an application example in which asimilar technology is adopted in edge emphasis, whereby edge components,as well as the noise components, are extracted from the subbandcoefficients resulting from the multiple resolution transformation, theextracted edge components are combined, a constant multiple factordependent upon the image structure is applied to the synthesized edgecomponent in the real spatial plane and then the product is added to theoriginal image.

Patent reference 1: U.S. Pat. No. 5,526,446Patent reference 2: U.S. Pat. No. 6,937,772Patent reference 3: U.S. Pat. No. 6,754,398

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

However, there is an issue yet to be properly addressed in the methodsdisclosed in all the patent reference literatures 1˜3 described above,in that since only a single type of noise removal processing is executedon a given subband plane resulting from the multiple resolutiontransformation, it is difficult to assure both accurate removal of allthe noise components present in the subband plane and reliable retentionof the effective image structure information pertaining to the subbandimage. Namely, if noise is removed to an optimal extent at which theimage structure remains intact, a residual noise component is bound toremain in a subband plane, whereas if the noise components arecompletely removed, the image structure is bound to be destroyed. Inshort, residual noise components occurring individually incorrespondence to various subbands cannot be accurately extracted evenby adopting the concept of the ultimate optimal noise removal rate forthe entire image disclosed in patent reference 3.

There is another issue, completely different from that discussed above,with regard to the art taught in reference 3, pertaining to the easewith which the noise components and the edge components are handled innoise processing and edge emphasis processing executed through the useof multiple resolutions. Namely, since the synthesized noise componentsand the synthesized edge components are limited to those having beenextracted in the high-frequency subbands constituting the completesystem in the multiple resolution system, the frequency characteristicsof the noise components and the edge components cannot be adjusted witha very high degree of freedom.

Means for Solving the Problems

According to the 1st aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which a plurality of band-limited images with resolutionsdecreasing in sequence are generated by filtering the input originalimage; a first noise removal step in which virtual noise removalprocessing is executed individually for each of the band-limited images;a second noise removal step in which actual noise removal processing isexecuted for the individual band-limited images based upon theband-limited images from which noise has been virtually removed throughthe first noise removal step; and an image acquisition step in which anoise-free image of the original image is obtained based upon theindividual band-limited images from which noise has been actuallyremoved through the second noise removal step, and the virtual noiseremoval processing executed in the first noise removal step and theactual noise removal processing executed in the second noise removalstep are differentiated in correspondence to a frequency band of aband-limited image.

According to the 2nd aspect of the present invention, in the imageprocessing method according to the 1st aspect, it is preferred thatnoise is removed to a greater extent in the virtual noise removalprocessing executed in the first noise removal step relative to theactual noise removal processing executed in the second noise removalstep.

According to the 3rd aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which a plurality of band-limited images with resolutionsdecreasing in sequence are generated by filtering the input originalimage; a first noise extraction step in which noise components containedin the band-limited images are individually extracted; a second noiseextraction step in which noise components to be reflected in theoriginal image are re-extracted from the individual band-limited imagesbased upon the noise components having been extracted from theband-limited images in the first noise extraction step; a noisesynthesis step in which the noise components having been re-extractedfrom the band-limited images in the second noise extraction step aresynthesized; and an image acquisition step in which a noise-free imageof the original image is acquired based upon the synthesized noisecomponent.

According to the 4th aspect of the present invention, in the imageprocessing method according to the 3rd aspect, it is preferred that thenoise components are intrinsically re-extracted from the individualband-limited images in correspondence to frequency bands of theband-limited images in the second noise extraction step.

According to the 5th aspect of the present invention, in the imageprocessing method according to the 4th aspect, it is preferred that:low-frequency images and high-frequency images with resolutionsdecreasing in sequence are generated in the multiple-resolution imagesgeneration step; and the noise components to be reflected in theoriginal image are re-extracted by weighting the noise components havingbeen extracted in the first noise extraction step differently betweenthe low-frequency images and the high-frequency images in the secondnoise extraction step.

According to the 6th aspect of the present invention, in the imageprocessing method according to the 4th aspect, it is preferred that thenoise components to be reflected in the original image are re-extractedby weighting the noise components having been extracted in the firstnoise extraction step differently between band-limited images withvarying levels of resolution in the second noise extraction step.

According to the 7th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which a plurality of band-limited images with resolutionsdecreasing in sequence are generated by filtering the input originalimage; a noise extraction step in which a noise component contained inone of the band-limited images is extracted and then a noise componentin another band-limited image is extracted in sequence based upon thenoise component having been extracted; and a noise removal step in whichnoise in the original image is removed based upon noise components inthe individual band-limited images having been extracted, and a signalintensity level with which the noise component extracted from oneband-limited image in the noise extraction step is reflected whenextracting the noise component from another band-limited image duringthe noise extraction step is set differently from a signal intensitylevel with which the noise component extracted from one band-limitedimage in the noise extraction step is reflected in the original imageduring the noise removal step.

According to the 8th aspect of the present invention, in the imageprocessing method according to the 7th aspect, it is preferred that thenoise component contained in the one band-limited image is extracted anda noise component contained in another band-limited image assuming adifferent resolution is extracted in sequence based upon the noisecomponent having been extracted in the noise extraction step.

According to the 9th aspect of the present invention, in the imageprocessing method according to the 7th aspect, it is preferred thatlow-frequency images and high-frequency images with resolutionsdecreasing in sequence are generated in the multiple-resolution imagesgeneration step.

According to the 10th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which sets of band-limited images each constituted with at leasttwo types of band-limited images are generated in sequence over aplurality of resolutions by filtering the input original image,generating a set of band-limited images constituted with at least twodifferent types of band-limited images and then repeatedly filtering atleast one type of band-limited images among the band-limited images; anoise extraction step in which processing whereby noise componentscontained in the two types of band-limited images are extracted and anoise component in at least one type of band-limited image at anotherresolution is extracted based upon the extracted noise components, isexecuted in sequence so as to extract noise components in the individualband-limited images at various resolutions; and a noise removal step inwhich noise in the original image is removed from based upon noisecomponents in the individual band-limited images having been extracted,and an inter-band signal intensity with which a set of noise componentsof at least two types of band-limited images obtained through the noiseextraction step is reflected when extracting the noise component inanother band-limited image in the noise extraction step is setdifferently from an inter-band signal intensity with which the set ofnoise components of at least two types of band-limited images obtainedthrough the noise extraction step is reflected in the original imageduring the noise removal step.

According to the 11th aspect of the present invention, in the imageprocessing method according to the 7th aspect, it is preferred that twodifferent types of images that are low-frequency images andhigh-frequency images are generated to make up the sets of band-limitedimages constituted with at least two different types of band-limitedimages in the multiple resolution image generation step.

According to the 12th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which one or more low-frequency images with resolutionsdecreasing in sequence and one or more high-frequency images with theresolutions decreasing in sequence are generated by decomposing theinput original image; a noise modulation step in which a noise componentcontained in each low-frequency image and a noise component contained ineach high-frequency image are individually extracted, a low-frequencynoise image and a high-frequency noise image are generated incorrespondence to the low-frequency image and the high-frequency imageand weights applied to the noise components over different frequencybands are modulated by applying a weighting coefficient to at leasteither the low-frequency noise image or the high-frequency noise imagehaving been generated; a noise synthesis step in which the low-frequencynoise image and the high-frequency noise image having undergone thenoise modulation step are combined to generate a single synthesizednoise image with higher resolution by one stage and noise images arerepeatedly synthesized in sequence until a single noise image signalwith a resolution matching the resolution of the original image isobtained; and a noise removal step in which noise contained in theoriginal image is removed based upon the synthesized noise image signal.

According to the 13th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which one or more low-frequency images having resolutionsdecreasing in sequence and one or more high-frequency images with theresolutions decreasing in sequence are generated by decomposing theinput original image; a noise extraction step in which a noise componentcontained in each low-frequency image and a noise component contained ineach high-frequency image are individually extracted and a low-frequencynoise image and a high-frequency noise image are generated incorrespondence to the low-frequency image and the high-frequency image;a noise modulation step in which weights applied to the noise componentsover different frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency noise image or thehigh-frequency noise image having been generated; a noise synthesis stepin which the low-frequency noise image and the high-frequency noiseimage having undergone the noise modulation step are combined togenerate a single synthesized noise image with higher resolution by onestage and the synthesized noise image with the higher resolution iscombined with the low-frequency noise image corresponding to thelow-frequency image with the higher resolution by one stage so as togenerate a new synthesized low-frequency noise image; a noise synthesisrepeating step in which synthesis is repeatedly executed by repeating asequence of the noise extraction step, the noise modulation step and thenoise synthesis step until a single noise image signal with a resolutionmatching the resolution of the original image is obtained; and a noiseremoval step in which noise contained in the original image is removedbased upon the synthesized noise image signal.

According to the 14th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which one or more low-frequency images with resolutionsdecreasing in sequence and one or more high-frequency images with theresolutions decreasing in sequence are generated by decomposing theinput original image; a noise extraction step in which a noise componentcontained in each low-frequency image and a noise component contained ineach high-frequency image are individually extracted and a low-frequencynoise image and a high-frequency noise image are generated incorrespondence to the low-frequency image and the high-frequency image;a noise modulation step in which weights applied to the noise componentsover different frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency noise image or thehigh-frequency noise image having been generated; a noise synthesis stepin which the low-frequency noise image and the high-frequency noiseimage having undergone the noise modulation step are combined togenerate a single synthesized noise image with higher resolution by onestage; and a noise removal step in which noise contained in the originalimage is removed based upon the synthesized noise image.

According to the 15th aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: an image input step in which an original image constituted ofa plurality of pixels is input; a multiple-resolution images generationstep in which one or more low-frequency images with resolutionsdecreasing in sequence and one or more high-frequency images with theresolutions decreasing in sequence are generated by decomposing theinput original image; a noise extraction step in which a noise componentcontained in each low-frequency image and a noise component contained ineach high-frequency image are individually extracted and a low-frequencynoise image and a high-frequency noise image are generated incorrespondence to the low-frequency image and the high-frequency image;a noise modulation step in which weights applied to the noise componentsover different frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency noise image or thehigh-frequency noise image having been generated; a noise synthesis stepin which the low-frequency noise image and the high-frequency noiseimage having undergone the noise modulation step are combined togenerate a single synthesized noise image with higher resolution by onestage and the synthesized noise image with the higher resolution iscombined with the low-frequency noise image corresponding to thelow-frequency image with the higher resolution by one stage so as togenerate a new synthesized low-frequency noise image; and a noiseremoval step in which noise contained in the original image is removedbased upon the synthesized noise image signal.

According to the 16th aspect of the present invention, in the imageprocessing method according to any one of the 12th through 15th aspects,it is preferred that the low-frequency noise image and thehigh-frequency noise image corresponding to the low-frequency image andthe high-frequency image are generated based upon observation of localsignal values in the low-frequency image and the high-frequency image inthe noise extraction step.

According to the 17th aspect of the present invention, an imageprocessing method adopted to emphasize edges in an image, comprises: animage input step in which an original image constituted of a pluralityof pixels is input; a multiple-resolution images generation step inwhich one or more low-frequency images with resolutions decreasing insequence and one or more high-frequency images with the resolutionsdecreasing in sequence are generated by decomposing the input originalimage; an edge component generation step in which an edge component isextracted by applying a band pass filter to each low-frequency image andan edge component is extracted by applying a band pass filter to eachhigh-frequency image and a low-frequency edge component image and ahigh-frequency edge component image are generated in correspondence tothe low-frequency image and the high-frequency image; an edge componentmodulation step in which weights applied to the edge components overdifferent frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency edge component image orthe high-frequency edge component image having been generated; an edgecomponent synthesis step in which the low-frequency edge component imageand the high-frequency edge component image having undergone the edgecomponent modulation step are combined to generate a single synthesizededge component image with higher resolution by one stage and edgecomponent images are repeatedly synthesized in sequence until a singleedge component image with a resolution thereof matching the resolutionof the original image is obtained; and an edge emphasis step in whichedges contained in the original image are emphasized based upon thesynthesized edge component image.

According to the 18th aspect of the present invention, an imageprocessing method adopted to emphasize edges in an image, comprises: animage input step in which an original image constituted of a pluralityof pixels is input; a multiple-resolution images generation step inwhich one or more low-frequency images having resolutions decreasing insequence and one or more high-frequency images with the resolutionsdecreasing in sequence are generated by decomposing the input originalimage; an edge component generation step in which an edge component isextracted by applying a band pass filter to each low-frequency image andan edge component is extracted by applying a band pass filter to eachhigh-frequency image and a low-frequency edge component image and ahigh-frequency edge component image are generated in correspondence tothe low-frequency image and the high-frequency image; an edge componentmodulation step in which weights applied to the edge components overdifferent frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency edge component image orthe high-frequency edge component image having been generated; an edgecomponent synthesis step in which the low-frequency edge component imageand the high-frequency edge component image having undergone the edgecomponent modulation step are combined to generate a single synthesizededge component image with a higher resolution by one stage; and an edgeemphasis step in which edges contained in the original image areemphasized based upon the synthesized edge component image.

According to the 19th aspect of the present invention, in the imageprocessing method according to any one of the 5th, 9th and 11th through18th aspects, it is preferred that the low-frequency image and thehigh-frequency image correspond to 1) a low-frequency component and ahigh-frequency component generated through orthogonal wavelettransformation, 2) a Gaussian component and a Laplacian component in aLaplacian pyramid representation, or 3) a low-frequency component andhigh-frequency components each corresponding to a specific direction indirectional wavelet transformation.

According to the 20th aspect of the present invention, in the imageprocessing method according to the 19th aspect, it is preferred thatwhen multiple-resolution images are generated through two-dimensionalorthogonal wavelet transformation, the low-frequency image correspondsto an LL subband and the high-frequency image corresponds to an LHsubband, an HL subband or an HH subband.

According to the 21st aspect of the present invention, an imageprocessing method adopted to remove noise present in an image,comprises: inputting an original image constituted of a plurality ofpixels; sequentially generating images with varying resolutions by thatthe input original image undergoes multiple resolution transformation;extracting a noise component by using an image generated at a givenresolution; using the noise component extracted at the given resolutionfor purposes of extracting a noise component at another resolution andsynthesizing a noise component to be removed from the original imagethrough inverse multiple resolution transformation; and applyingdifferent weights to the noise component extracted at the givenresolution to be used for purposes of extracting the noise component atthe other resolution and to the noise component extracted at the givenresolution to be used to synthesize the noise component to be removedfrom the original image through inverse multiple resolutiontransformation.

According to the 22nd aspect of the present invention, an imageprocessing program enables a computer or an image processing apparatusto execute an image processing method according to any one of the 1stthrough the 21st aspects.

According to the 23rd aspect of the present invention, an imageprocessing apparatus comprises an image processing program according tothe 22nd aspect installed therein.

ADVANTAGEOUS EFFECT OF THE INVENTION

The present invention, assuming the structure described above, allowsnoise to be removed for noise extraction with a high level of freedom byan extent matching the required intensity without being restricted bythe conditions for retaining the image structure intact, so as to assureaccurate noise extraction, and also allows the original image structureto be retained intact, by assuring preservation of the image structure.

BRIEF DESCRIPTION OF THE DRAWINGS

(FIG. 1) The image processing apparatus achieved in an embodiment of thepresent invention

(FIG. 2) A flowchart of the color space conversion processing executedby the personal computer 1

(FIG. 3) A flowchart of the luminance component processing executed in afirst embodiment

(FIG. 4) A flowchart of the chrominance component processing executed inthe first embodiment

(FIG. 5) The subband division achieved through five-stage wavelettransformation

(FIG. 6) The simplest Laplacian filter commonly in use

(FIG. 7) The weighting coefficients for the low-frequency subband (LL)and the high-frequency subbands (LH, HL, HH) of the noise component inthe luminance component

(FIG. 8) The weighting coefficients for the low-frequency subband (LL)and the high-frequency subbands (LH, HL, HH) of the noise component in achrominance component

(FIGS. 9A-9C) A setting screen that may be brought up when settingintensity parameters (intensity) σth, rth, a frequency characteristicsadjustment parameter (graininess) k0 and a parameter (sharpness) λrelated to the noise removal intensity

(FIG. 10) A flowchart of the luminance component processing executed ina second embodiment

(FIG. 11) A flowchart of the chrominance component processing executedin the second embodiment

(FIG. 12) A flowchart of the edge emphasis processing using multipleresolution transformation

(FIG. 13) The weighting coefficients for the low-frequency subband (LL)and the high-frequency subbands (LH, HL, HH) of the edge component inthe luminance component

(FIG. 14) A schematic diagram illustrating the relationship between thelow-frequency subband and the high-frequency subbands in various typesof multiple-resolution representations

(FIG. 15) A schematic illustration of frequency bands covered in thehigh-frequency subbands and the low-frequency subband in amultiple-resolution representations

(FIG. 16) A flowchart of luminance component processing similar to thatshown in FIG. 3, except that specific noise removal processing on theimage signals in the real space is skipped

(FIG. 17) A flowchart of luminance component processing similar to thatshown in FIG. 10, except that specific noise removal processing on theimage signals in the real space is skipped

(FIG. 18) A flowchart of edge emphasis processing using multipleresolution transformation, similar to that shown in FIG. 12 except thatspecific edge component extraction processing on the image signals inthe real space is skipped

(FIG. 19) The structure adopted in a personal computer

(FIG. 20) A flowchart of the noise removal processing executed to removenoise in the luminance component (luminance signal) in a variation ofthe first embodiment

BEST MODE FOR CARRYING OUT THE INVENTION Basic Concept

First, the rationale and the reasoning for the need to adopt thealgorithms to be described in reference to the embodiments and the basicconcept of the method to be adopted in conjunction with the algorithmsare explained.

As described above, the technologies proposed in the related art, whichuse multiple-resolution representations, are divided into two primarygroups. There are various methods of multiple-resolutionrepresentations, such as Steerable wavelet transformation and DCTpyramid representation as well as orthogonal wavelet transformation andLaplacian-pyramid representation. Since the relationship with which theindividual methods correspond to one another has been made clear in thereferences of the known art, an explanation is given on a singleexample, i.e., orthogonal wavelet transformation, for purposes ofsimplification.

In a first method, noise is removed from high-frequency-side subbands(LH, HL, HH) resulting from the orthogonal wavelet transformation. In asecond method, noise is removed sequentially from the low-frequencysubband (LL) resulting from the orthogonal wavelet transformation.

Noise is normally removed from a color image by dividing the color imageinto a luminance (brightness) plane and chrominance (color difference)planes and individually removing noise from the luminance plane and thechrominance planes. Grainy noise is reduced by removing noise from theluminance plane, whereas color mottling noise is reduced by removingnoise from the chrominance planes.

The following has become clear as a result of a test conducted byadopting these two different types of algorithms in conjunction with acolor image expressed in luminance-chrominance representation. Namely,both effective color mottling noise removal and desirable colorstructure preservation are assured when removing noise from thechrominance components through the method whereby noise is sequentiallyremoved from the low-frequency subband rather than through the methodwhereby noise is removed from the high-frequency subbands. In otherwords, unlike noise removal from the high-frequency-side subbands in thechrominance components, which is problematic in that it readily inducesbleeding at color boundaries, noise removal from the low-frequency-sidesubbands does not readily induce bleeding at color boundaries.

At the same time, it has been learned that noise can be removed from theluminance component more effectively through noise removal from thehigh-frequency subbands rather than through the method whereby noise isremoved sequentially from the low-frequency component. In other words,the sequential noise removal from the low-frequency subband in theluminance component is problematic in that the gradation can be easilylost and a flat image that looks “binary image” may be created. Thenoise removal from the high-frequency subbands, on the other hand,assures gradation retention and preserves the image structure such asthe texture of the image in a desirable manner.

This characteristic difference between the luminance component and thechrominance components is assumed to be attributable to the differencebetween the frequency characteristics of the image structure on theluminance plane and the frequency characteristics of the image structureon the chrominance planes, which necessitates the noise components inthe luminance plane and the chrominance planes to be extracted indifferent frequency spaces.

Accordingly, the noise removal for the luminance component was executedby removing noise from the high-frequency subbands as in the related artand the noise removal for the chrominance components was executed bysequentially removing noise from the low-frequency subband as in therelated art. The results of this experiment indicated that no matter howsuperior an edge-preserving smoothing filter may be used as each noiseremoval filter, noise components manifesting as streaks or checkeredpatterns will not be removed over flat areas in the luminance componentand color noise manifesting as points is likely to remain to asignificant extent, particularly around color boundary areas in thechrominance components.

FIG. 15 schematically illustrates the frequency bands covered by thehigh-frequency subbands and the low-frequency subband in amultiple-resolution representation. In reference to FIG. 15, the issuesdiscussed above are further examined, first with regard to the luminancecomponent. Since the original image can be completely reconstructedsimply by expressing it with a low-frequency subband corresponding tothe lowest resolution and high-frequency subbands corresponding to theindividual resolutions, the noise component over the full frequency bandwould appear to be covered at a superficial level simply through thenoise removal from the high-frequency subbands alone. However, as noiseis sequentially removed from the high-frequency components at varyinglevels of resolution, the noise component may not be extracted to thefull extent over areas where the intensity of the frequency bandsoverlapping each other in different resolutions is low.

Likewise, the noise component over the full frequency band would appearto be covered at a superficial level with regard to the chrominancecomponent processing, simply as the noise in the low-frequency subbandalone is removed. However, point-like noise are primarily detected ashigh-frequency component-side signals as the original image is brokendown to the low-frequency component and the high-frequency components,allowing the noise component entering the high-frequency component side,to remain.

The opposite characteristics discussed above can be assumed to cause thedifference in the optimal frequency space for the luminance componentnoise extraction and the chrominance component noise extraction. Namely,the findings obtained through the conducted testing substantiate thefact that the smoothing filtering processing executed on the real spaceplane handled via a single channel or on low-frequency side subbands ina multiple-resolution representation handled via multiple channels islikely to lead to gradation loss whereby the gradations are adjusted toa uniform level matching the average within the filtering range.

This fact leads to the conclusion that the majority of the edgecomponent in the image structure is reflected in the luminancecomponent, with a great deal of noise component, too, also readilydirected to the high-frequency subband side. Under such circumstances,the noise component cannot be extracted successfully on thelow-frequency subband side and loss of gradation characteristics occursreadily.

The chrominance components, on the other hand, are generally regarded toreadily reflect an image component representing overall colorinformation with a gentle behavioral pattern over a wide range. Thechrominance components are thus normally assumed not to containsignificant color texture that changes drastically. Accordingly, thenoise component can be separated from the chrominance components moreeasily on the low-frequency side, unlike the noise component in theluminance component, which can be extracted more easily on thehigh-frequency side. However, as the noise component fluctuationinformation readily flows into the high-frequency subbands as well, asis widely known, and a target original image may contain significantcolor texture, noise component separation on the high-frequency subbandside is also likely to be necessary.

These issues are addressed in the embodiment by extracting noisecomponents both from the high-frequency subbands and from thelow-frequency subband so as to pick up the residual noise component froma subband which has a conjugate relation to the subbands initially usedfor the noise removal. The conjugate subband is the low-frequencysubband when removing the noise in the luminance component but is ahigh-frequency subband when removing the noise in a chrominancecomponent.

However, as has been confirmed through testing, noise removal from theconjugate subband is likely to result in image structure destruction asexplained earlier and thus, successful noise removal cannot be achievedby simply adopting the method described above. Accordingly, in theconcept adopted in the embodiment, the noise component extraction andthe actual noise removal are regarded as separate processes and the roleof the conjugate subband in the actual noise removal is basicallyregarded as a complementary role, so as to prevent destruction of theimage structure.

Namely, in the actual noise removal, noise is removed from the luminancecomponent by assigning the high-frequency side subbands as primary bandsand assigning the low-frequency side subband as a complementary band,whereas noise is removed from the chrominance component by assigning thelow-frequency side subband as a primary band and assigning thehigh-frequency side subbands as complementary bands. However, if ahigh-performance noise removal filter is used in the chrominancecomponent noise removal, the roles of the primary band and thecomplementary bands do not need to be as distinguishable as those of theprimary bands and the complementary band used in the luminance componentnoise removal and the levels of their efficacy may be substantiallyequal to each other, according to the findings obtained through testing.The rationale for this may be that the difference of the overallcharacteristics between the luminance plane and the chrominance planeintegrating the difference of image structure characteristics and thenoise component flow characteristics with which the noise componentflows between the various bands, implies the presence of differentfrequency projection spaces optimal for the luminance plane noiseremoval and the chrominance plane noise removal.

However, if the extent of noise removal from a complementary band is setat a somewhat low level, the residual noise component will not be fullyextracted even from the complementary band. Accordingly, the roleassignment to the individual subbands should be adjusted in the actualnoise removal. According to the present invention, the noise extractionand the noise removal are conceptualized separately, allowing forflexible thinking in that as long as a subband image is used solely forpurposes of noise extraction, the subband image can be virtuallydestroyed as much as needed down to the level at which accurate noiseextraction is enabled. Namely, in the concept adopted in the presentinvention, there are two different types of noise removal, i.e., virtualnoise removal executed for noise extraction and noise removal executedas the actual noise removal processing.

Through the measures described above, an environment, in which theresidual streaking noise in the luminance component can be extractedeasily as data clearly distinguishable from the image structure data inthe low-frequency image, is created and an environment in which theresidual point-like noise in the chrominance components can be easilyextracted as data clearly distinguishable from the image structure datain a high-frequency image is created.

In order to enable more accurate noise extraction via virtual noiseremoval, a method of extracting noise components codependent on oneanother at varying resolutions, instead of extracting the noisecomponent independently of one another from the individual subbandplanes corresponding to the low-frequency image and the high-frequencyimage, is adopted. Namely, noise is fully removed from an upper layersubband image or a lower layer subband image at different resolutions toachieve a noise-free state even if it means the image structure isvirtually destroyed, the results of the noise removal are reflected onthe subband at the currently targeted resolution and noise is extractedsequentially as the resolution is switched.

The method for extracting noise from the low-frequency side subband bysequentially switching the resolution has already been disclosed in therelated art in U.S. Pat. No. 6,937,772 and Japanese Laid Open PatentPublication No. 2000-224421. However, according to the presentinvention, the concept is also adopted in noise removal mainly from thehigh-frequency side subbands or from both the low-frequency side and thehigh-frequency side. Specific methods with which the art may beeffectively adopted under such circumstances will be described later inreference to embodiments.

The sequential noise removal is advantageous in that when it is adoptedin noise removal from both the high-frequency band and the low-frequencyband, the noise extraction performance in the complementary band issignificantly enhanced. Namely, noise manifesting as vertical/horizontalstreaks or a checkered pattern present in the low-frequency sidecomplementary band can be completely extracted through the sequentialnoise removal adopted in conjunction with the luminance component,whereas point-like noise present in the high-frequency sidecomplementary band can be extracted fully through the sequential noiseremoval adopted in conjunction with the chrominance components.

In a sense, the use of orthogonal wavelet transformation with a lesserovercompleteness of processing as a secondary two-dimensional separationfilter also indirectly contributes to the specific characteristics ofthe residual noise component in the luminance component, manifesting asthe vertical/horizontal streaks or checkered patterns. In order toeliminate any such specific directionality, Steerable wavelets may beutilized so as to generate high-frequency bands in correspondence tomultiple directions in the multiple resolution transformation.

However, since high-frequency bands are generated in correspondence tomultiple directions, a greater number of planes will need to beprocessed for noise removal, requiring a much greater memory capacity tohold data used in the processing. In other words, the Steerable waveletmethod entailing a great increase in the processing load cannot beadopted with ease. As an alternative technology, the sequential noiseremoval, which can be adopted with ease and simplicity, effectivelyaddresses the issues discussed earlier, and its effectiveness is furtherenhanced when adopted both in conjunction with both the low-frequencysubband and the high-frequency subbands.

However, the advantages of the present invention, described in referenceto the embodiments, are not limited to noise removal through theorthogonal wavelet transformation and they will be also effective innoise removal in an image in multiple-resolution representations such asthe Laplacian-pyramid representation and the Steerable wavelettransformation since the technology according to the present inventionsuccessfully complements the weaknesses in the multiple resolutiontransformation filter characteristics of the Laplacian-pyramidrepresentation and the Steerable wavelet transformation.

Virtual noise removal may be sequentially reflected among variousresolutions by disintegrating the resolution toward the lower side or byintegrating the resolution toward the higher side. The former method isreferred to as “sequential analysis” and the latter is referred to as“sequential synthesis” in the description of embodiments.

The term “analysis” in this context implies breaking down image data tomultiple resolution data at lower resolutions, whereas the term“synthesis” in this context is equivalent to integrating the broken downmultiple resolution data into the initial high-resolution data(synthesizing the initial high-resolution data). In the case of wavelettransformation, the term “analysis” refers to the wavelet transformationand the term “synthesis” refers to inverse wavelet transformation. Amethod that may be adopted in the “sequential analysis” is described inthe first embodiment and a method that may be adopted in the “sequentialsynthesis” is described in reference to the second embodiment.

FIRST EMBODIMENT

FIG. 1 shows the image processing apparatus achieved in an embodiment ofthe present invention. The image processing apparatus is constitutedwith a personal computer 1. The personal computer 1, which is connectedwith a digital camera 2, a recording medium 3 such as a CD-ROM, anothercomputer 4 and the like, is able to receive various types of image data.The personal computer 1 executes the image processing to be explainedbelow on the image data provided thereto. The personal computer 1 isconnected to the computer 4 via an electronic communication network 5,which may be the Internet.

The program that enables the computer 1 to execute the image processingis provided in a recording medium such as a CD-ROM or by anothercomputer via the Internet or another electronic communication networkconnected to the personal computer in a manner similar to that shown inFIG. 1, and the program thus provided is installed within the personalcomputer 1. FIG. 19 shows the structure adopted in the personal computer1. The personal computer 1 comprises a CPU 11 and its peripheralcircuits 13. The installed program is executed by the CPU.

The program to be provided via the Internet or another electroniccommunication network is converted to and transmitted as a signal on acarrier wave which is carried through the electronic communicationnetwork, i.e., a transmission medium. Namely, the program can bedistributed as a computer-readable computer program product adopting anyof various modes including a recording medium and a carrier wave.

The following is an explanation of the image processing executed by thepersonal computer 1. FIG. 2 presents a flowchart of the image processingexecuted by the personal computer 1 in the first embodiment. In step S1,linear RGB image data are input. In step S2, the input image data areconverted to data in a uniform color and uniform noise space. In stepS3, the data undergo noise removal processing. In step S4, the colorspace data undergo reverse conversion. In step S5, the image data havingundergone the processing are output. The following is a detailedexplanation of the processing executed in each step.

(1) Color Space Conversion

In step S1, RGB color image data with linear gradation with respect tothe light intensity are input. In step S2, the input image data areconverted to data in a uniform noise space in which the noise is set tothe uniform level relative to the gradations, so as to create anenvironment in which noise removal is facilitated. In the embodiment,the image data are converted to data in a uniform color and uniformnoise space so as to further assure color uniformity as well as noiseuniformity, thereby providing both the noise removal effect and thecolor reproducibility retention effect.

Such a uniform color and uniform noise space used as the imageprocessing space is described in Japanese Patent Application No.2004-365881 submitted by the inventor of the present invention andaccordingly, Japanese Patent Application No. 2004-365881 should bereferred to for details of the uniform color and uniform noise space.The following explanation is provided by assuming that the target inputdata are sRGB image data. It is to be noted, however, that gammacorrection, having been executed on the sRGB image should first beundone, to revert to the initial image with linear gradation beforestarting the processing.

First, the linear gradation RGB values are converted to XYZ values.Namely, the image data are converted to the XYZ calorimetric systemspace. This processing is executed through 3×3 matrix conversion, whichis defined in correspondence to the RGB original stimulus spectralcharacteristics. For instance, the data constituting the input sRGBimage may undergo the following standardized conversion.

X=0.4124*R+0.3576*G+0.1805*B  (1)

Y=0.2126*R+0.7152*G+0.0722*B  (2)

Z=0.0193*R+0.1192*G+0.9505*B  (3)

Next, the data in the XYZ space are converted to a nonlinear gradationL̂âb̂ space representing a perceptive attribute with a pseudo-uniformcolor distribution. In this description, the term “L̂âbb̂ space” is usedto refer to a variation of the uniform color space L*a*b* in the relatedart achieved by modifying the L*a*b* space so as to assume noiseuniformity.

L̂=100*f(Y/Y0)  (4)

â=500*[f(X/X0)−f(Y/Y0)]  (5)

b̂=200*[f(Y/Y0)−f(Z/Z0)]  (6)

X0, Y0 and Z0 in the expressions above each represent a value determinedin correspondence to the illuminating light. For instance, X0, Y0 and Z0may assume values 95.045, 100.00 and 108.892 respectively in a 2° visualfield under standard light D65. In addition, the nonlinear gradationconversion function f(t) is defined below. Noise uniformity is achievedbased upon the characteristics of the function f(t). It is to be notedthat the variable t, expressed as; t=(Y/Y0), t=(X/X0), t=(Z/Z0), assumesa value normalized based upon the maximum value representing the numberof gradations for the X, Y and Z values so that the relationshipsexpressed as 0≦(Y/Y0)≦1, 0≦(X/X0)≦1, 0≦(Z/Z0)≦1, are satisfied.

[Expression 1]

ƒ(t)=t+√{square root over (t+ε)}  (7)

If necessary, the origin point and the saturation point may benormalized by using the following expression.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack & \; \\{{f(t)} = \frac{\sqrt{t + ɛ} - \sqrt{ɛ}}{\sqrt{1 + ɛ} - \sqrt{ɛ}}} & (8)\end{matrix}$

ε in the expression above represents an offset signal applied to thelinear gradation signal. While ε assumes a value depending on thespecific sensor in use, a value very close to zero, for instance, willbe assumed when a low sensitivity setting is selected and a valueapproximately equal to 0.05 will be assumed when a high sensitivitysetting is selected.

(2) Noise Removal

The noise removal processing executed in step S3 is now explained. FIG.3 presents a flowchart of the processing executed for the luminance(brightness) component (luminance signal), whereas FIG. 4 presents aflowchart of the processing executed for a chrominance (colordifference) component (color different signal). However, FIG. 4 simplypresents the part of the chrominance component processing that differsfrom the luminance component processing in FIG. 3, as detailed later.

(2-1) Multiple Resolution Transformation

While FIGS. 3 and 4 illustrate multiple resolution (multiresolution)transformation achieved through five-stage wavelet transformation, thenumber of stages over which wavelet transformation is executed may beadjusted in correspondence to the size of the input original image. Aslong as the wavelet transformation is executed over approximately fivestages, the frequency band of the target noise component can besubstantially covered under normal circumstances.

(2-1-1) Wavelet Transformation: Analysis/Decomposition Process

In the wavelet transformation, through which image data are transformedinto frequency component data, the frequency component in the image isdivided into a high-pass component and a low-pass component. In theembodiment, a 5/3 filter is utilized to execute the five-stage wavelettransformation mentioned above. The 5/3 filter generates the low-passcomponent with a filter having five taps (one-dimension/5 pixels) andgenerates the high-pass component with a filter having three taps(one-dimension/3 pixels).

The high-pass component and the low-pass component are generated asexpressed below. In the expressions, n indicates the pixel position andx[ ] indicates the pixel value assumed at the target image undergoingthe wavelet transformation. For instance, n assumes a value in the rangeof 0˜49 if 100 pixels are set along the horizontal direction. Byextracting the high-pass component and the low-pass component asexpressed below, the high-pass component and the low-pass component dataat 50 pixels, i.e., half the pixels, are extracted.

High-pass component: d[n]=x[2n+1]−(x[2n+2]+x[2n])/2  (9)

Low-pass component: s[n]=x[2n]+(d[n]+d[n−1])/4  (10)

The one-dimensional wavelet transformation defined as described above isexecuted along the horizontal direction and the vertical directionindependently through two-dimensional separation filtering processing soas to achieve wavelet decomposition. The coefficient s is directed ontothe L plane, whereas the coefficient d is directed onto the H plane. Thereal space plane, identical to the input image, is designated as an LL0plane. The LL0 plane is handled as the low-frequency subband with thehighest resolution as well as the wavelet transformation coefficientlow-frequency subband LL1, LL2, LL3, LL4 and LL5.

More specifically, the five-stage wavelet transformation is executedsequentially as indicated below by using the expressions presentedearlier. In the embodiment, the wavelet transformation is executed asnoise signals are extracted sequentially by using the LL component dataand the LH, HL and HH component data generated at the individual stages,as described later. It is to be noted that LL component data arelow-frequency subband data, whereas the LH, HL and HH data arehigh-frequency subband data. Also, a low-frequency subband may bereferred to as a low-frequency image and a high-frequency subband may bereferred to as a high-frequency image. Furthermore, each subband may bereferred to as a frequency band-limited image. A low-frequency subbandis an image with band limits imposed upon the frequency bands of theoriginal image toward the low-frequency side, whereas a high-frequencysubband is an image with band limits imposed upon the frequency band ofthe original image toward the high-frequency side.

First-stage wavelet transformation: LL0 (real space)→LL1, LH1, HL1, HH1Second-stage wavelet transformation: LL1→LL2, LH2, HL2, HH2Third-stage wavelet transformation: LL2→LL3, LH3, HL3, HH3Fourth stage wavelet transformation: LL3→LL4, LH4, HL4, HH4Fifth-stage wavelet transformation: LL4→LL5, LH5, HL5, HH5

FIG. 5 shows the subband partition achieved through the five-stagewavelet transformation. For instance, through the first-stage wavelettransformation, high-pass component data and low-pass component data areextracted from the image data in all the rows extending along thehorizontal direction in the real space. As a result, high-pass componentdata and low-pass component data corresponding to half the entire numberof pixels are extracted along the horizontal direction. The extractedhigh-pass component data and low-pass component data may then be storedinto, for instance, memory areas on the right side and on the left sideof the memory area where the image data in the real space have beenpresent.

Next, high-pass component data and low-pass component data are extractedas expressed earlier, individually from the high-pass component datahaving been stored on the right side in the memory area and from thelow-pass component data having been stored on the left side in thememory area, along all the columns extending in the vertical direction.As a result, high-pass component data and low-pass component data areextracted both from the high-pass component data stored on the rightside in the memory area and from the low-pass component data stored onthe left side in the memory area. The high-pass component data and thelow-pass component data thus extracted are stored on the bottom side andthe top side respectively in the memory area where the correspondingsource data have been present.

HH indicates data extracted as high-pass component data along thevertical direction from data having been extracted as high-passcomponent data along the horizontal direction, HL indicates dataextracted as low-pass component data along the vertical direction fromthe data having been extracted as high-pass component data along thehorizontal direction, LH indicates data extracted as high-pass componentdata along the vertical direction from data having been extracted aslow-pass component data along the horizontal direction and LL indicatesdata extracted as low-pass component data along the vertical directionfrom the data having been extracted as low-pass component data along thehorizontal direction. However, since the processing along the verticaldirection and the processing along the horizontal direction are executedindependently, the same processing results are obtained by extractingthe data in the reverse order.

Next, through the second-stage wavelet transformation, high-passcomponent data and low-pass component data are extracted in a similarmanner from the data LL extracted through the first-stage wavelettransformation as low-pass component data along the vertical directionfrom the data having been extracted as the low-pass component data alongthe horizontal direction. The partition shown in FIG. 5 is achieved byrepeatedly executing the processing described above over five stages.

(2-1-2) Inverse Wavelet Transformation: Synthesis/Reconstruction Process

The inverse wavelet transformation (inverse multiple resolutiontransformation) is executed as expressed below.

x[2n]=s[n]−(d[n]+d[n−1])/4  (11)

x[2n+1]=d[n]+(x[2n+2]+x[2n])/2  (12)

It is to be noted that as shown in FIG. 3, a signal expressing the imageis input to be used as the value x for the wavelet transformation, thenoise components contained in the generated wavelet transformationcoefficients s and dare extracted and a noise image x is generated bysubstituting the extracted noise components for s and d used in theinverse wavelet transformation.

(2-2) Noise Removal Processing

Any noise removal filter may be used in the noise removal processingexecuted on the individual subband planes. Typical examples ofedge-preserving smoothing filters include a filters such as thatdescribed in reference “Jong—Sen Lee, “Digital image smoothing and theSigma filter”, Computer Vision, Graphics and Image Processing 24 (1983)pp 255˜269” and bilateral filters such as that described in reference“C. Tomasi et al., “Bilateral Filtering for Gray and Color Images,”Proceedings of the 1998 IEEE International Conference on ComputerVision, Bombay, India”.

However, in this embodiment there are shown, for example, a higherperformance modified bilateral filter (see Japanese Patent ApplicationNo. 2004-367263 submitted by the inventor of the present invention fordetails) and a simpler and faster noise removal filter (see JapanesePatent Application No. 2005-101545 submitted by the inventor of thepresent invention. The noise removal methods achieved through the use ofthis filter is to be referred to as a Laplacian noise extractionmethod). Either of these noise removal filters may be used.

V(vector r) represents an original signal in the input subband imageplane whereas V′(vector r) or V″(vector r) represents a signal on animage plane having undergone noise removal. It is to be noted that rwith an arrow (referred to as vector r) and r′ with an arrow (referredto as vector r′) in the expression below, each representing a vector,indicate two dimensional positional coordinates.

(2-2-1) Modified Bilateral Filter

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 3} \right\rbrack & \; \\{{V^{\prime}\left( \overset{\rightarrow}{r} \right)} = \frac{\int_{{{\overset{\rightarrow}{r}}^{\prime}} < {2r_{th}}}{{V\left( {\overset{\rightarrow}{r}}^{\prime} \right)}\exp \left\{ {{- \frac{{{{V\left( {\overset{\rightarrow}{r}}^{\prime} \right)} - {V\left( \overset{\rightarrow}{r} \right)}}}^{2}}{\sigma_{th}^{2}}} \cdot \frac{{{{\overset{\rightarrow}{r}}^{\prime} - \overset{\rightarrow}{r}}}^{2}}{r_{th}^{2}}} \right\} \ {{\overset{\rightarrow}{r}}^{\prime}}}}{\int_{{{\overset{\rightarrow}{r}}^{\prime}} < {2r_{th}}}{\exp \left\{ {{- \frac{{{{V\left( {\overset{\rightarrow}{r}}^{\prime} \right)} - {V\left( \overset{\rightarrow}{r} \right)}}}^{2}}{\sigma_{th}^{2}}} \cdot \frac{{{{\overset{\rightarrow}{r}}^{\prime} - \overset{\rightarrow}{r}}}^{2}}{r_{th}^{2}}} \right\} \ {{\overset{\rightarrow}{r}}^{\prime}}}}} & (13)\end{matrix}$

The threshold value rth taken along the spatial direction should assumea value within a range of approximately 0.5˜3.0 pixels so as to createan overlap of hierarchical layers with different resolutions, since thenoise removal filter assumes a range approximately twice the thresholdvalue. The threshold value may also be adjusted in correspondence to theimage-capturing sensitivity. The threshold value σth taken along thegradation direction should assume a greater value as a higherimage-capturing sensitivity level is selected and the optimal valueshould be adjusted in correspondence to each subband as well.

The filter weighting coefficient in the bilateral filter in the relatedart is represented by the product of the weighting coefficient w_photo[V′−V] of the photometric term that takes the difference (V′−V) betweenpixel values alone as an argument and the weighting coefficientw_geometric[r′−r] of the geometric term which takes the spatial distance(r′−r) alone as an argument. In this sense, the bilateral filter in therelated art may be referred to as a separately weighted bilateralfilter, the weighting coefficient of which can be separated into thephotometric term and the geometric term. The weighting coefficient ofthe modified bilateral filter, on the other hand, cannot be separatedinto the photometric term and the geometric term. In other words, it isan integrated weighted bilateral filter, the weighting coefficient ofwhich is expressed with a single exponential function, taking on anexponent assuming the value matching the product of the two arguments.

(2-2-2) Laplacian Noise Extraction Method

Noise is extracted from the chrominance component data as expressedbelow

[Expression 4]

V′({right arrow over (r)})=V({right arrow over (r)})−∇² V({right arrowover (r)})·ƒ(∇² V({right arrow over (r)}))  (14)

Noise is extracted from the luminance component data as expressed below

[Expression 5]

V′({right arrow over (r)})=V({right arrow over (r)})−∇² V({right arrowover (r)})·ƒ(∇² V({right arrow over (r)}))  (15)

V″({right arrow over (r)})=V′({right arrow over (r)})+∇² V′({right arrowover (r)})·ƒ(∇² V′({right arrow over (r)}))  (16)

f(x) is a function expressed below. ∇² represents the Laplacian filter(high-pass filter). FIG. 6 shows the simplest Laplacian filter in commonuse.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 6} \right\rbrack & \; \\{{f(x)} = {\exp\left( {- \frac{x^{2}}{\sigma_{th}^{2}}} \right)}} & (17)\end{matrix}$

It is to be noted that the threshold value σth taken along the gradationdirection should be set based upon a concept similar to that applied tothe modified bilateral filter detailed above. It will be obvious thatindividually selected optimal values should be set as the thresholdvalue for the luminance component data and the chrominance componentdata.

It is to be noted that the modified bilateral filter and the Laplacianfilter described above are each a function of the values of signalscontained in localized areas. Namely, noise is extracted by checking thevalues indicated in the signals in the individual localized areas in thelow-frequency subband and the high-frequency subbands.

(2-3) Noise Removal from the Luminance Component (L̂)

Next, in reference to FIG. 3, the noise removal from the luminancecomponent data (L̂) is described in detail. As explained earlier, thenoise is extracted through the “sequential analysis”. It is to be notedthat the individual processing phases (x-x) described below areindicated as (x-x) in correspondence in FIG. 3.

(2-3-1) Multiple Resolution Transformation and Sequential NoiseExtraction (2-3-1-1) Processing at the Highest Resolution in the RealSpace

In processing (0-1), noise is removed from an image signal S0(LL0) inthe real space through the noise removal filter described above, so asto generate a noise-free image signal S0′(LL0). In processing (0-2), thenoise component in the LL0 subband is extracted as expressed;n0(LL0)=S0(LL0)−S0′(LL0). In processing (0-3), the noise signal n0(LL0)retaining the initial intensity (or multiplied by a factor of α(0)) issubtracted from the image signal S0(LL0), thereby removing the noisefrom S0(LL0). It is to be noted that 0<α(0)≦1 and that α(0) normallytakes on the value of 1. In processing (0-4), the image signal in theLL0 plane, having undergone noise removal through the processing (0-3),undergoes wavelet transformation, thereby generating image signalsS1(LL1, LH1, HL1, HH1) at ½ resolution.

(2-3-1-2) Processing Executed at ½ Resolution

In processing (1-1), noise is removed from the individual image signalsS1(LL1, LH1, HL1, HH1) via the noise removal filter described above and,as a result, noise-free image signals S1′(LL1, LH1, HL1, HH1) aregenerated. In processing (1-2), the noise components in the individualsubbands are extracted as expressed; n1(LL1)=S1(LL1)−S1′(LL1),n1(LH1)=S1(LH1)−S1′(LH1), n1(HL1)=S1(HL1)−S1′(HL1) andn1(HH1)=S1(HH1)−S1′(HH1). In processing (1-3), the noise signal n1(LL1)retaining the initial intensity (or multiplied by a factor of α(1)) issubtracted from the image signal S1(LL1), thereby removing the noisefrom S1(LL1). It is to be noted that 0<α(1)≦1 and that α(1) normallytakes on the value of 1. In processing (1-4), the image signal in theLL1 plane, having undergone noise removal through the processing (1-3),undergoes wavelet transformation, thereby generating image signalsS2(LL2, LH2, HL2, HH2) at ¼ resolution.

(2-3-1-3) Processing Executed at ¼ Resolution

This processing is executed much the same way as the processing executedat the ½ resolution described in (2-3-1-2) above.

(2-3-1-4) Processing Executed at ⅛ Resolution

This processing is executed much the same way as the processing executedat the ½ resolution described in (2-3-1-2) above.

(2-3-1-5) Processing Executed at 1/16 Resolution

In processing (4-1), noise is removed from the individual image signalsS4(LL4, LH4, HL4, HH4) via the noise removal filter described above and,as a result, noise-free image signals S4′(LL4, LH4, HL4, HH4) aregenerated. In processing (4-2), the noise components in the individualsubbands are extracted as expressed; n4(LL4)=S4(LL4)−S4′(LL4),n4(LH4)=S4(LH4)−S4′(LH4), n4(HL4)=S4(HL4)−S4′(HL4) andn4(HH4)=S4(HH4)−S4′(HH4). In processing (4-3), the noise signal n4(LL4)retaining the initial intensity (or multiplied by a factor of α(4)) issubtracted from the image signal S4(LL4), thereby removing the noisefrom S4(LL4). It is to be noted that 0<α(4)≦1 and that α(4) normallytakes on the value of 1. In processing (4-4), the image signal in theLL4 plane, having undergone noise removal through the processing (4-3),undergoes wavelet transformation, thereby generating image signalsS5(LL5, LH5, HL5, HH5) at 1/32 resolution.

(2-3-1-6) Processing Executed at 1/32 Resolution, i.e., the LowestResolution

In processing (5-1), noise is removed from the individual image signalsS5(LL5, LH5, HL5, HH5) via the noise removal filter described above and,as a result, noise-free image signals S5′(LL5, LH5, HL5, HH5) aregenerated. In processing (5-2), the noise components in the individualsubbands are extracted as expressed; n5(LL5)=S5(LL5)−S5′(LL5),n5(LH5)=S5(LH5)−S5′(LH5), n5(HL5)=S5(HL5)−S5′(HL5) andn5(HH4)=S5(HH5)−S5′(HH5).

The processing described above differs from the related art in that thenoise components in the high-frequency subbands LH, HL and HH on thelow-resolution side generated from a low-frequency subband LL havingundergone the sequential noise removal, too, are extracted with a highlevel of accuracy after the subbands data first undergo thehigh-resolution-side noise removal. Namely, the results of the noiseremoval executed on an upper layer low-frequency subband affect noiseextraction from the lower layer high-frequency subbands as well as thelower layer low-frequency subband. Thus, the noise components can beextracted both from the low-frequency subband and the high-frequencysubbands, containing very little residual noise to begin with, in themultiple-resolution representations.

(2-3-2) Noise Component Frequency Characteristics Adjustment

Next, the extracted noise components are modified into noise componentsto be used for the actual noise removal. This modification is achievedby re-extracting noise components for the actual noise removal from theextracted noise components. This process, executed to assure theindestructibly of the luminance component image structure, fulfills therole of a variable parameter with which the visual effect achievedthrough noise removal can be adjusted with ease. Namely, the noisecomponent frequency characteristics are adjusted by altering the weightapplied to the low-frequency subband (LL) and the high-frequencysubbands (LH, HL, HH) relative to each other. The parameter may beprovided as a graininess-modifying parameter for the noise removal in agraphic user interface in software processing or the like. In otherwords, different weighting coefficients are applied to the noisecomponent in a low-frequency subband and the noise components in thehigh-frequency subbands (e.g., k0 applied to the LL subband and 1applied to the other subbands in the example presented below, so as tomodulate the weights applied to different noise component frequencybands.

This process is executed as expressed below and the individualprocessing procedures correspond to the processing (0-5), the processing(1-5), the processing (2-5), the processing (3-5), the processing (4-5)and the processing (5-5) in FIG. 3.

n0′(LL0)=k0(0)*n0(LL0)  (18)

n1′(LL1)=k0(1)*n1(LL1)  (19)

n2′(LL2)=k0(2)*n2(LL2)  (20)

n3′(LL3)=k0(3)*n3(LL3)  (21)

n4′(LL4)=k0(4)*n4(LL4)  (22)

n5′(LL5)=k0(5)*n5(LL5)  (23)

During the processing, the following measures are taken.

n1′(LL1) and n1(LH1, HL1, HH1) are directly bundled together inton1′(LL1, LH1, HL1, HH1).n2′(LL2) and n2(LH2, HL2, HH2) are directly bundled together inton2′(LL2, LH2, HL2, HH2).n3′(LL3) and n3(LH3, HL3, HH3) are directly bundled together inton3′(LL3, LH3, HL3, HH3).n4′(LL4) and n4(LH4, HL4, HH4) are directly bundled together inton4′(LL4, LH4, HL4, HH4).n5′(LL5) and n5(LH5, HL5, HH5) are directly bundled together inton5′(LL5, LH5, HL5, HH5).

Under normal circumstances, k0(0), k0(1), k0(2), k0(3), k0(4) and k0(5)are all set equal to one another at k0, which is variable within a rangeof 0≦k0≦1. k0 should assume a value close to the median value, e.g.,0.5, in order to prevent manifestation of significant residual noisecomponents and also preserve textural structure in the image byretaining the optimal level of graininess, whereas a lower value such as0.2 may be assumed for k0 if it is more important to preserve the imagestructure by sustaining a higher level of graininess. If, on the otherhand, the wide range high-frequency background noise over the entireimage plane needs to be eliminated, a higher value, e.g., 0.8, should beassumed for k0.

The noise signal in a high-frequency subband sustaining the initialintensity should be directly output under normal circumstances. In otherwords, the weight applied to the high-frequency subbands should be setto a greater value than the weight applied to the low-frequency subband.However, under certain circumstances, the high-frequency subbands noisesignal may be multiplied by a weighting coefficient. FIG. 7 shows theweighting coefficients applied to the low-frequency subband (LL) and thehigh-frequency subbands (LH, HL, HH).

As described above, the present invention adopts the noise removalconcept that assumes two different types of noise removal, i.e., thenoise removal executed for noise component extraction and the noiseremoval executed for actual noise removal during which the imagestructure must remain intact. Noise removal for noise extraction can beexecuted with as much intensity, as needed, unrestricted by theconditions imposed to assure image structure preservation. Namely, theintensity with which noise is removed for noise component extraction canbe increased freely, unlike the intensity with which the noise removalfor actual noise removal is executed. Consequently, noise can beextracted accurately from each subband while assuring a satisfactorylevel of image structure preservation.

In addition, the frequency characteristics of the synthesized noisecomponent can be adjusted with ease simply by applying a weightingcoefficient to a complementary subband among the high-frequency subbandsand the low-frequency subband. Thus, an environment in which the visualeffect achieved through noise removal can be adjusted with ease whileassuring fine noise removal is created. In addition, since the need tore-execute the noise removal processing for noise extraction, which isnormally the most time-consuming, is eliminated, the results achieved byadjusting the visual effect can be obtained promptly.

(2-3-3)

Synthesis of Noise Components

The modified noise components are then synthesized through inversewavelet transformation executed in sequence from the lowest resolutionside.

(2-3-3-1) Processing Executed at 1/32 Resolution, i.e., the LowestResolution

In processing (5-7), the noise signals n5′(LL5, LH5, HL5, HH5)corresponding to the single layer, having undergone the inter-bandweighting processing, undergo inverse wavelet transformation, so as togenerate a noise signal N5(LL4) corresponding to the LL4 subband plane.

(2-3-3-2) Processing Executed at 1/16 Resolution

In processing (4-6), the noise signal n4′(LL4) having been extractedfrom the LL4 subband plane and having undergone the weightingprocessing, is combined with N5(LL4) through addition processingexpressed as below

n4″(LL4)=n4′(LL4)+N5(LL4)  (24).

n4″(LL4) and n4′(LH4, HL4, HH4) are directly bundled together inton4″(LL4, LH4, HL4, HH4). Through this processing, the noise componentcorresponding to the LL4 plane is generated by combining the noisecomponents over the two layers, as indicated in FIG. 3. It is to benoted, however, that the noise components for LH4, HL4 and HH4correspond to a single layer. In processing (4-7), the noise signalsn4″(LL4, LH4, HL4, HH4) obtained by combining the noise components overthe two layers, undergo inverse wavelet transformation, so as togenerate a noise signal N4(LL3) corresponding to the LL3 subband plane.(2-3-3-3) Processing Executed at ⅛ Resolution

This processing is executed in much the same way as the “processingexecuted at 1/16 resolution” described in (2-3-3-2) above.

(2-3-3-4) Processing Executed at ¼ Resolution

This processing is executed in much the same way as the “processingexecuted at 1/16 resolution” described in (2-3-3-2) above.

(2-3-3-5) Processing Executed at ½ Resolution

In processing (1-6), the noise signal n1′(LL1) having been extractedfrom the LL1 subband plane and having undergone the weightingprocessing, is combined with N2(LL1) through addition processingexpressed as below.

n1″(LL1)=n1′(LL1)+N2(LL1)  (25)

n1″(LL1) and n1′(LH1, HL1, HH1) are directly bundled together inton1″(LL1, LH1, HL1, HH1). In processing (1-7), the noise signals n″(LL1,LH1, HL1, HH1) obtained by combining the noise components over the twolayers, undergo inverse wavelet transformation, so as to generate anoise signal N1(LL0) corresponding to the LL0 subband plane.

(2-3-3-6) Processing Executed at the Highest Resolution in the RealSpace

In processing (0-6), the noise signal n0′(LL0) having been extractedfrom the LL0 subband plane and having undergone the weightingprocessing, is combined with N1(LL0) through addition processingexpressed as below.

n0″(LL0)=n0′(LL0)+N1(LL0)  (26).

It is to be noted that the processing according to the present inventiondescribed above differs from the related art in that the noise componentfor the low-frequency subband is generated through noise synthesis bycombining the noise components over two layers, i.e., the noisecomponent obtained by synthesizing the noise components in thelow-frequency subband and the high-frequency subbands on the lowresolution side and the noise component extracted from the low-frequencysubband at the target resolution are combined. Through these measures,the correct noise component minus any residual noise component can besynthesized with ease, while assuring a high level of image structurepreservation and noise characteristics that allow the visual effect tobe adjusted easily.

The adjustability of the frequency characteristics may be furtherincreased by selecting different levels of intensity for the noisecomponents in the varying resolution levels when adding together thenoise components over the two layers. Under such circumstances, theprocessing should be executed as expressed below.

n4″(LL4)=n4′(LL4)+β(5)*N5(LL4)  (27)

n3″(LL3)=n3′(LL3)+β(4)*N4(LL3)  (28)

n2″(LL2)=n2′(LL2)+β(3)*N3(LL2)  (29)

n1″(LL1)=n1′(LL1)+β(2)*N2(LL1)  (30)

n0″(LL0)=n0′(LL0)+β(1)*N1(LL0)  (31)

It is to be noted that 0<β(1)≦1, 0<β(2)≦1, 0<β(3)≦1, 0<β(4)≦1 and0<β(5)≦1. The use of such a parameter may prove effective when, forinstance, random noise cannot be assumed to be white noise, whichremains uniform over the full frequency range.

(2-3-4) Actual Noise Removal Processing

The actual noise removal is executed on the single synthesized noisecomponent with the resolution matching that in the real space, to whicha noise removal factor, i.e., a weighting coefficient parameter λ, hasbeen applied so as to adjust the extent of noise removal for the overallimage. Namely, the processing is executed as expressed below.

S0NR(LL0)=S0(LL0)−λ*n0″(LL0)  (32)

with parameter λ assuming value within the range of 0≦λ≦1.(2-4) Noise Removal from the Chrominance Component (â).

As does the luminance component data (L̂), the chrominance component data(â) undergoes noise extraction through the “sequential analysis”. Thenoise removal processing executed for the chrominance component differsfrom the noise removal from the luminance component in that theweighting coefficients are applied to different target subbands in orderto modify the frequency characteristics from the target subbands in theprocessing executed to adjust the noise component frequencycharacteristics described in (2-3-2) above, i.e., different weightingprocessing is executed and in that the noise removal factor parameter isselected differently from the noise removal factor parameter set in the“Actual noise removal processing” described in (2-3-4) above. Thefollowing explanation focuses on these differences. It is to be notedthat FIG. 4 only shows the part of the “noise component frequencycharacteristics adjustment” processing that is different from theprocessing shown in FIG. 3.

(2-4-1) Noise Component Frequency Characteristics Adjustment

As expressed below, a weighting coefficient parameter used to assureboth a desirable point-like noise removal effect and color retentionwhen actually removing noise from the chrominance component data isapplied to the noise components in the high-frequency subbands (LH, HL,HH), since the low-frequency subband is assigned as a primary band andthe high-frequency subbands are assigned as complementary bands in thecase of the chrominance component data.

n1′(LH1)=k1(1)*n1(LH1)  (33)

n1′(HL1)=k1(1)*n1(HL1)  (34)

n1′(HH1)=k2(1)*n1(HH1)  (35)

n2′(LH2)=k1(2)*n2(LH2)  (36)

n2′(HL2)=k1(2)*n2(HL2)  (37)

n2′(HH2)=k2(2)*n2(HH2)  (38)

n3′(LH3)=k1(3)*n3(LH3)  (39)

n3′(HL3)=k1(3)*n3(HL3)  (40)

n3′(HH3)=k2(3)*n3(HH3)  (41)

n4′(LH4)=k1(4)*n4(LH4)  (42)

n4′(HL4)=k1(4)*n4(HL4)  (43)

n4′(HH4)=k2(4)*n4(HH4)  (44)

n5′(LH5)=k1(5)*n5(LH5)  (45)

n5′(HL5)=k1(5)*n5(HL5)  (46)

n5′(HH5)=k2(5)*n5(HH5)  (47)

During the processing, the following measures are taken.

n1(LL1) and n1′(LH1, HL1, HH1) are directly bundled together inton1(LL1, LH1, HL1, HH1).n2(LL2) and n2′(LH2, HL2, HH2) are directly bundled together inton2′(LL2, LH2, HL2, HH2).n3(LL3) and n3′(LH3, HL3, HH3) are directly bundled together inton3′(LL3, LH3, HL3, HH3).n4(LL4) and n4′(LH4, HL4, HH4) are directly bundled together inton4′(LL4, LH4, HL4, HH4).n5(LL5) and n5′(LH5, HL5, HH5) are directly bundled together inton5′(LL5, LH5, HL5, HH5).

Under normal circumstances, k1(1), k1(2), k1(3), k1(4) and k1(5) are allset equal to one another at k1 and k2(1), k1(2), k2(3), k2(4) and k2(5)are all set equal to one another at k2. k1 and k2 are variable within arange of 0≦k1, k2≦1 and may be set to, for instance, k1=0.9 and k2=0.8.Under normal operating conditions, they should both be set to valueswithin a range of 0.8˜1.0. In addition, while k1 is commonly set for theLH subband and the HL subband, different values may be selected for theLH subband and the HL subband instead. FIG. 8 shows the weightingcoefficients applied to the low-frequency subband (LL) and thehigh-frequency subbands (LH, HL, HH). The weighting coefficient appliedto the low-frequency subband (LL) is 1 and accordingly, the valueinitially assumed in the noise signal is directly used. In other words,the weight applied to the low-frequency subband should be set to agreater value than the weights applied to the high-frequency subbands.However, the weights applied to the low-frequency subband and thehigh-frequency subbands are very nearly the same, as k1 is set to 0.9and k2 is set to 0.8, both close to 1.

(2-4-2) Actual Noise Removal Processing

The actual noise removal processing for the chrominance component datais executed much the same way as the actual noise removal processingexecuted for the luminance component data (L̂) described in (2-3-4).However, the noise removal factor λ set for the chrominance componentmay assume the value of 1.0 under normal circumstances.

By taking full advantage of the characteristics of the multi-channelrepresentation as described above, the noise extraction processing isexecuted in the optimal frequency spaces where the noise components canbe extracted with ease, based upon the difference of the imagestructures and the difference of the noise characteristics betweenseparated luminance and chrominance planes. As a result, fine noiseremoval from a color image is achieved with ease while assuring aminimum loss of image structures and minimum residual noise.

(2-5) Noise Removal from the Chrominance Component (b̂)

The noise removal for the chrominance component (b̂) is executed in muchthe same way as the noise removal for the chrominance component (â)described in (2-4).

The following three different functions are fulfilled via the main noiseremoval parameters that can be adjusted by the user with ease insoftware or the like.

(1) An intensity parameter (intensity) used for noise componentextraction: σth (rth as well, in some filters)(2) A frequency characteristics adjustment parameter (graininess)related to the noise graininess: k0(3) A parameter (sharpness) related to the noise removal intensity: λ

FIGS. 9A-9C show a setting screen that may be brought up to allow theuser to set the intensity parameters (intensity) σth and rth, thefrequency characteristics adjustment parameter (graininess) k0 and thenoise removal intensity parameter (sharpness) λ. A slide bar isdisplayed in correspondence to each parameter so as to allow the user toselect a preferred value for a given parameter by setting the pointer inthe slide bar to a specific position.

More specifically, the setting screen in FIGS. 9A-9C is brought up ondisplay at the monitor (not shown) of the personal computer 1 and theuser sets the pointer in each slide bar to a specific position via akeyboard (not shown) or a mouse (not shown). The user is thus able toset the parameters with ease. For instance, the user may adjust thefrequency characteristics adjustment parameter (graininess) k0 asexplained earlier, so as to alter the visual effect to be achievedthrough noise removal with ease while preserving fine imagecharacteristics. In addition, the operation will be executed in quickresponse as soon as k0 and λ are changed.

(3) Reverse Color Space Conversion, Image Data Output

In step S4 in FIG. 2, the image data resulting from the noise removalprocessing executed in step S3 described above undergo conversionprocessing which is the reverse of “(1) color space conversion” havingbeen executed in step S2 so as to convert the image data back to dataconstituting an RGB image. In step S5, the image data constituting theRGB image resulting from the reverse conversion are output.

As described above, in the first embodiment, two types of processingequivalent to two different types of noise removal, are executed eachfor noise extraction or noise removal, and the noise removal resultsobtained by removing noise from an upper layer low-frequency subband aremade to affect the noise extraction from the lower layer high-frequencysubbands as well as the noise extraction from the lower layerlow-frequency subband. Namely, noise is extracted sequentially from boththe high-frequency subbands and the low-frequency subband constitutingimages obtained through multiple resolution transformation whileallowing the processing of either type of subbands to be influenced bythe characteristics of the other type of subbands. Consequently, thelevel of freedom with which different frequency bands can be combined inthe processing increases and the noise extraction can be executed in thefrequency space optimized for the noise extraction. This means that finenoise removal is enabled without loss of image structure whileminimizing the residual noise that is not extracted.

Namely, fine noise removal processing (edge-preserving smoothing)assuring a high level of image structure preservation while effectivelyaddressing the issue of residual noise in a regular image such as adigital picture, is realized through the embodiment.

It is to be noted that in the embodiment, a specific type of noiseremoval processing is executed on the image signal S0(LL0) in the realspace (see FIG. 3). However, a load of the processing executed on theimage signal S0(LL0) in the real space is bound to be extremely heavy.Even through the processing executed at lower resolutions equal to orlower than that corresponding to the image signals S1(LL1, LH1. HL1,HH1) in the embodiment alone, a sufficient level of fine noise removalprocessing is achieved. Accordingly, the specific noise removalprocessing executed on the image signal S0(LL0) in the real space may beskipped. FIG. 16 presents a flowchart of the processing executed on theluminance component (luminance signal) by skipping the specific noiseremoval processing on the image signal S0(LL0) in the real space in FIG.3. The chrominance components, too, may be processed in a similarmanner. Through these measures, fine noise removal processing can beexecuted with a lighter processing load.

SECOND EMBODIMENT

In reference to the first embodiment, a method that may be adopted in a“sequential analysis” whereby noise is extracted sequentially as theimage data at a higher resolution are decomposed to data at a lowerresolution. Now, in reference to the second embodiment, a method thatmay be adopted in a “sequential synthesis” whereby noise is extractedsequentially as data having been analyzed through multiple resolutiontransformation are integrated so as to synthesize image data toward thehigher resolution is explained.

Since the image processing apparatus in the second embodiment assumes astructure similar to that of the image processing apparatus in the firstembodiment illustrated in FIG. 1, its explanation is omitted. Inaddition, the overall flow of the image processing executed by thepersonal computer 1 in the second embodiment, too, is similar to that inthe flowchart presented in FIG. 2 and, accordingly, its explanation isomitted. The following explanation focuses on the aspects of the secondembodiment that differ from those of the processing executed in thefirst embodiment.

(1) Color Space Conversion (2) Noise Removal (2-1) Multiple ResolutionTransformation (2-1-1) Wavelet Transformation: Analysis/DecompositionProcess (2-1-2) Inverse Wavelet Transformation: Synthesis/ReconstructionProcess (2-2) Noise Removal Processing (2-2-1) Modified Bilateral Filter(2-2-2) Laplacian Noise Extraction

Since details of the processes listed above are similar to those in thefirst embodiment, their explanation is omitted.

(2-3) Noise Removal from the Luminance Component (L̂)

FIG. 10 presents a flowchart of the processing executed for luminancecomponent data, whereas FIG. 11 presents a flowchart of the processingexecuted for chrominance component data. It is to be noted that FIG. 11simply shows the part of the chrominance component processing that isdifferent from the luminance component processing in FIG. 10. It is alsoto be noted that the individual processing phases (xx) and (xx-x)described below are also indicated as (xx) and (xx-x) in FIG. 10.

(2-3-1) Multiple Resolution Transformation (2-3-1-1) Processing Executedat the Highest Resolution in the Real Space

In processing (10), the image signal S0(LL0) on the real space planeundergoes wavelet transformation so as to generate image signals S1(LL1,LH1, HL1, HH1) at ½ resolution.

(2-3-1-2) Processing Executed at ½ Resolution

In processing (11), the image signal S1(LL1) on the LL1 plane undergoeswavelet transformation so as to generate image signals S1(LL2, LH2, HL2,HH2) at ¼ resolution.

(2-3-1-3) Processing Executed at ¼ Resolution

In processing (12), the image signal S2(LL2) on the LL2 plane undergoeswavelet transformation so as to generate image signals S3(LL3, LH3, HL3,HH3) at ⅛ resolution.

(2-3-1-4) Processing Executed at ⅛ Resolution

In processing (13), the image signal S3(LL3) on the LL3 plane undergoeswavelet transformation so as to generate image signals S4(LL4, LH4, HL4,HH4) at 1/16 resolution.

(2-3-1-5) Processing Executed at 1/16 Resolution

In processing (14), the image signal S4(LL4) on the LL4 plane undergoeswavelet transformation so as to generate image signals S5(LL5, LH5, HL5,HH5) at 1/32 resolution.

(2-3-2) Sequential Noise Extraction

(2-3-2-1) Processing Executed at 1/32 Resolution i.e., at the LowestResolution

In processing (15-1), noise is removed from the individual image signalsS5(LL5, LH5, HL5, HH5) and, as a result, noise-free image signalsS5′(LL5, LH5, HL5, HH5) aregenerated. In processing (15-2), the noisesignals in the individual subbands are extracted as expressed;n5(LL5)=S5(LL5)−S5′(LL5), n5(LH5)=S5(LH5)−S5′(LH5),n5(HL5)=S5(HL5)−S5′(HL5) and n5(HH5)=S5(HH5)−S5′(HH5). In processing(15-4) the noise signals n5(LL5, LH5, HL5, HH5) undergo inverse wavelettransformation (synthesis), thereby generating a noise signal N5(LL4) tobe used for noise extraction in correspondence to the LL4 subband plane.

(2-3-2-2) Processing Executed at 1/16 Resolution

In processing (14-0), the noise signal N5(LL4) retaining the initialintensity (or multiplied by a factor of α(5)) is subtracted from theimage signal S4(LL4), so as to obtain an image signal S4′(LL4). It is tobe noted that 0<α(5)≦1 and that α(5) normally takes on the value of 1.It is to be noted that S4′(LL4) and S4(LH4, HL4, HH4) are directlybundled together into S4′(LL4, LH4, HL4, HH4).

In processing (14-1), noise is removed from the individual image signalsS4′(LL4, LH4, HL4, HH4) and, as a result, noise-free image signalsS4″(LL4, LH4, HL4, HH4) are generated while these signals are indicatedas S4″(LL4″, LH4′, HL4′, HH4′) in FIG. 10, they are identical toS4″(LL4, LH4, HL4, HH4). In processing (14-2), the noise signals in theindividual subbands are extracted as expressed;n4(LL4)=S4′(LL4)−S4″(LL4), n4(LH4)=S4′(LH4)−S4″(LH4),n4(HL4)=S4′(HL4)−S4″(HL4) and S4′ n4(HH4)=S4′(HH4)−S4″(HH4).

In processing (14-3), the noise signal n4(LL4) extracted through thenoise removal processing executed for the LL4 plane and the noise signalN5(LL4) obtained by synthesizing the noise components from the lowerlayer for noise extraction are combined through addition processingexecuted as expressed below.

n4′(LL4)=n4(LL4)+N5(LL4)  (48)

n4′(LL4) and n4(LH4, HL4, HH4) are directly bundled together inton4′(LL4, LH4, HL4, HH4). In processing (14-4), the noise signalsn4′(LL4, LH4, HL4, HH4) undergo inverse wavelet transformation togenerate a noise signal N4(LL3) corresponding to the LL3 subband plane.

(2-3-2-3) Processing Executed at ⅛ Resolution

This processing is executed much the same way as the processing executedat the 1/16 resolution, as described in (2-3-2-2) above.

(2-3-2-4) Processing Executed at ¼ Resolution

This processing is executed much the same way as the processing executedat the 1/16 resolution, as described in (2-3-2-2) above.

(2-3-2-5) Processing Executed at ½ Resolution

In processing (11-0), the noise signal N2(LL1) retaining the initialintensity (or multiplied by a factor of α(2)) is subtracted from theimage signal S1(LL1), so as to obtain an image signal S1′(LL1). It is tobe noted that 0<α(2)≦1 and that α(2) normally takes on the value of 1.It is also to be noted that S1′(LL1) and S1(LH1, HL1, HH1) are directlybundled together into S1′(LL1, LH1, HL1, HH1).

In processing (11-1), noise is removed from the individual image signalsS1′(LL1, LH1, HL1, HH1) and, as a result, noise-free image signalsS1″(LL1, LH1, HL1, HH1) are generated while these signals are indicatedas S1″(LL1″, LH1′, HL1′, HH1′) in FIG. 10, they are identical toS1″(LL1, LH1, HL1, HH1). In processing (11-2), the noise signals in theindividual subbands are extracted as expressed;n1(LL1)=S1′(LL1)−S1″(LL1), n1(LH1)=S1′(LH1)−S1″(LH1),n1(HL1)=S1′(HL1)−S1″(HL1) and n1(HH1)=S1′(HH1)−S1″(HH1).

In processing (11-3), the noise signal n1(LL1) extracted through thenoise removal processing executed for the LL1 plane and the noise signalN2(LL1) obtained by synthesizing the noise components from the lowerlayer for noise extraction are combined through addition processingexecuted as expressed below.

n1′(LL1)=n1(LL1)+N2(LL1)  (49)

n1′(LL1) and n1(LH1, HL1, HH1) are directly bundled together inton1′(LL1, LH1, HL1, HH1). In processing (11-4), the noise signalsn1′(LL1, LH1, HL1, HH1) undergo inverse wavelet transformation togenerate a noise signal N1(LL0) corresponding to the LL0 subband plane.

(2-3-2-6) Processing Executed at the Highest Resolution in the RealSpace

In processing (10-0), the noise signal N1(LL0) retaining the initialintensity (or multiplied by a factor of α(1)) is subtracted from theimage signal S0(LL0), so as to obtain an image signal S0′(LL0). It is tobe noted that 0<α(1)≦1 and that α(1) normally takes on the value of 1.In processing (10-1), noise is removed from the individual image signalS0′(LL0) and, as a result, a noise-free image signal S0″(LL0) isgenerated. In processing (10-2), the noise signal is extracted asexpressed; n0(LL0)=S0′(LL0)−S0″(LL0).

The feature of the embodiment that is particularly noteworthy is thatthe effect of the noise removal from the low resolution-sidehigh-frequency subbands, as well as the effect of the noise removal fromthe low resolution side low-frequency subband is reflected in theexecution of noise extraction from the higher resolution-sidelow-frequency subband. Namely, the results of noise removal from thelower layer low-frequency subband and the lower layer high-frequencysubbands together affect the noise extraction from the upper layerlow-frequency subband. Through these measures, the correct noisecomponent that needs to be extracted from the low-frequency subband sidein the multiple-resolution representations can be extracted, and thus, anoise component without significant residual noise is extracted.

By adopting such a “sequential synthesis” when processing the luminancecomponent data in particular, the noise removal effect achieved on thehigh-frequency subband side can be used to successfully draw out theresidual noise component manifesting as streaks or checkered patterns inthe low-frequency side subband.

(2-3-3) Noise Component Frequency Characteristics Adjustment

Next, the extracted noise components are modified into noise componentsto be used for the actual noise removal. Namely, the noise componentfrequency characteristics are adjusted by selecting different settingfor the weight applied to the low-frequency subband (LL) and the weightapplied to the high-frequency subbands (LH, HL, HH). The noise componentfrequency characteristics are adjusted based upon a principal identicalto that of the first embodiment and the parameter should be selected forthe processing in the second embodiment in much the same way as in thefirst embodiment.

The processing is executed as expressed below and the individualprocessing procedures correspond to processing (10-5), processing(11-5), processing (12-5), processing (13-5), processing (14-5) andprocessing (15-5) in FIG. 10.

n0″(LL0)=k0(0)*n0(LL0)  (50)

n1″(LL1)=k0(1)*n1(LL1)  (51)

n2″(LL2)=k0(2)*n2(LL2)  (52)

n3″(LL3)=k0(3)*n3(LL3)  (53)

n4″(LL4)=k0(4)*n4(LL4)  (54)

n5″(LL5)=k0(5)*n5(LL5)  (55)

During the processing, the following measures are taken.

n1″(LL1) and n1(LH1, HL1, HH1) are directly bundled together inton1″(LL1, LH1, HL1, HH1).n2″(LL2) and n2(LH2, HL2, HH2) are directly bundled together inton2″(LL2, LH2, HL2, HH2).n3″(LL3) and n3(LH3, HL3, HH3) are directly bundled together inton3″(LL3, LH3, HL3, HH3).n4″(LL4) and n4(LH4, HL4, HH4) are directly bundled together inton4″(LL4, LH4, HL4, HH4).n5″(LL5) and n5(LH5, HL5, HH5) are directly bundled together inton5″(LL5, LH5, HL5, HH5).

(2-3-4) Synthesis of Noise Components

The modified noise components are then synthesized through inversewavelet transformation executed in sequence from the lowest resolutionside. The synthesized noise component is to be used for the actual noiseremoval.

(2-3-4-1) Processing Executed at 1/32 Resolution, i.e., the LowestResolution

In processing (15-7), the noise signals n5″(LL5, LH5, HL5, HH5)corresponding to the single layer, having undergone the inter-bandweighting processing, undergo inverse wavelet transformation, so as togenerate a noise signal N5′(LL4) to be used for the actual noise removalin correspondence to the LL4 subband plane.

(2-3-4-2) Processing Executed at 1/16 Resolution

In processing (14-6), the noise signal n4″(LL4) having been extractedfrom the LL4 subband plane and having undergone the weightingprocessing, is combined with noise signal N5′(LL4) obtained bysynthesizing the noise components in the lower layer for actual noiseremoval through addition processing expressed as below.

n4′″(LL4)=n4″(LL4)+N5′(LL4)  (56)

n4′″(LL4) and n4″(LH4, HL4, HH4) are directly bundled together inton4′″(LL4, LH4, HL4, HH4). Through this processing, the noise componentcorresponding to the LL4 plane is generated by combining the noisecomponents over the two layers, as indicated in FIG. 10. It is to benoted, however, that the noise components for LH4, HL4 and HH4correspond to a single layer. In processing (14-7), the noise signalsn4′″(LL4, LH4, HL4, HH4), obtained by combining the noise componentsover the two layers, undergo inverse wavelet transformation, so as togenerate a noise signal N4′(LL3) corresponding to the LL3 subband plane.(2-3-4-3) Processing Executed at ⅛ Resolution

This processing is executed in much the same way as the “processingexecuted at 1/16 resolution” described in (2-3-4-2) above.

(2-3-4-4) Processing Executed at ¼ Resolution

This processing is executed in much the same way as the “processingexecuted at 1/16 resolution” described in (2-3-4-2) above.

(2-3-4-5) Processing Executed at ½ Resolution

In processing (11-6), the noise signal n1″(LL1) having been extractedfrom the LL1 subband plane and having undergone the weightingprocessing, is combined with N2′(LL1) obtained by synthesizing the noisecomponents from the lower layer for actual noise removal throughaddition processing expressed as below.

n1′″(LL1)=n1″(LL1)+N2′(LL1)  (57)

n1′″(LL1) and n1″(LH1, HL1, HH1) are directly bundled together inton1′″(LL1, LH1, HL1, HH1). In processing (11-7), the noise signalsn1′″(LL1, LH1, HL1, HH1) obtained by combining the noise components overthe two layers, undergo inverse wavelet transformation, so as togenerate a noise signal N1′(LL0) corresponding to the LL0 subband plane.

(2-3-4-6) Processing Executed at the Highest Resolution in the RealSpace

In processing (10-6), the noise signal n0″(LL0), having been extractedfrom the LL0 subband plane and having undergone the weightingprocessing, is combined with N1′(LL0) obtained by synthesizing the noisecomponents from the lower layer for actual noise removal throughaddition processing expressed as below.

n0′″(LL0)=n0″(LL0)+N1′(LL0)  (58)

The adjustability of the frequency characteristics may be furtherincreased by selecting different levels of intensity for the noisecomponents in varying resolutions when adding together the noisecomponents over the two layers, as in the first embodiment. Likewise,under such circumstances, the processing should be executed as expressedbelow.

n4′″(LL4)=n4″(LL4)+β(5)*N5′(LL4)  (59)

n3′″(LL3)=n3″(LL3)+β(4)*N4′(LL3)  (60)

n2′″(LL2)=n2″(LL2)+β(3)*N3′(LL2)  (61)

n1′″(LL1)=n1″(LL1)+β(2)*N2′(LL1)  (62)

n0′″(LL0)=n0″(LL0)+β(1)*N1′(LL0)  (63)

It is to be noted that 0<β(1)≦1, 0<β(2)≦1, 0<β(3)≦1, 0<β(4)≦1 and0<β(5)≦1.

A noteworthy feature of this processing is that two separate systems ofnoise synthesizing means are provided to generate two different types ofnoise components, one for noise extraction and the other for actualnoise removal, through synthesis. As a result, the optimizationprocessing executed to adjust the noise component intensitycharacteristics or adjust the noise component frequency characteristicsin correspondence to a given application is facilitated.

In addition, as in the first embodiment, the noise component for thelow-frequency subband is integrated through the noise synthesisprocessing by using noise components obtained from two different layers,i.e., the noise component made up with the noise components from boththe low-frequency subband and the high-frequency subbands on the lowresolution side and the noise component obtained from the low-frequencysubband at the target resolution itself, which clearly differentiatesthe present invention from the related art. Consequently, the noisefrequency characteristics adjustment is facilitated and noise componentsoptimized for the individual purposes of use of the two separate systemscan be combined.

(2-3-5) Actual Noise Removal Processing

The processing is executed in much the same way as “(2-3-4) Actual noiseremoval processing” in the first embodiment.

(2-4) Noise Removal from the Chrominance Component (â)

The processing is executed in much the same way as “(2-4) noise removalfrom the chrominance component (â)” in the first embodiment. However,the expressions used in the processing are defined slightly differently,and accordingly, they need to be modified as indicated below.

(2-4-1) Noise Component Frequency Characteristics Adjustment

n1″(LH1)=k1(1)*n1(LH1)  (64)

n1″(HL1)=k1(1)*n1(HL1)  (65)

n1″(HH1)=k2(1)*n1(HH1)  (66)

n2″(LH2)=k1(2)*n2(LH2)  (67)

n2″(HL2)=k1(2)*n2(HL2)  (68)

n2″(HH2)=k2(2)*n2(HH2)  (69)

n3″(LH3)=k1(3)*n3(LH3)  (70)

n3″(HL3)=k1(3)*n3(HL3)  (71)

n3″(HH3)=k2(3)*n3(HH3)  (72)

n4″(LH4)=k1(4)*n4(LH4)  (73)

n4″(HL4)=k1(4)*n4(HL4)  (74)

n4″(HH4)=k2(4)*n4(HH4)  (75)

n5″(LH5)=k1(5)*n5(LH5)  (76)

n5″(HL5)=k1(5)*n5(HL5)  (77)

n5″(HH5)=k2(5)*n5(HH5)  (78)

During the processing, the following measures are taken.

n1(LL1) and n1″(LH1, HL1, HH1) are directly bundled together inton1″(LL1, LH1, HL1, HH1).n2(LL2) and n2″(LH2, HL2, HH2) are directly bundled together inton2″(LL2, LH2, HL2, HH2).n3(LL3) and n3″(LH3, HL3, HH3) are directly bundled together inton3″(LL3, LH3, HL3, HH3).n4(LL4) and n4″(LH4, HL4, HH4) are directly bundled together inton4″(LL4, LH4, HL4, HH4).n5(LL5) and n5″(LH5, HL5, HH5) are directly bundled together inton5″(LL5, LH5, HL5, HH5).(2-5) Noise Removal from the Chrominance Component (b̂)

The processing is executed in much the same way as “(2-4) noise removalfrom the chrominance component (â)”.

As described above, in the second embodiment, two types of processing,equivalent to two different types of noise removal, are executed eachfor purposes of noise extraction or noise removal, and the noise removalresults of the noise removal from the lower layer high-frequencysubbands as well as the results of the noise removal from the lowerlayer low-frequency subband are allowed to affect the execution of thenoise extraction from the low-frequency subband in the upper layer.Namely, noise is extracted sequentially from both the high-frequencysubbands and the low-frequency subband constituting images obtainedthrough multiple resolution transformation while allowing the processingof either type of subband to be influenced by the characteristics of theother type of subband, as in the first embodiment. Consequently, thelevel of freedom with which different frequency bands can be combined inthe processing increases and the noise removal can be executed in thefrequency spaces optimized for the noise extraction. This means thatfine noise removal is enabled without loss of image structure whileminimizing the residual noise that is not extracted.

Namely, fine noise removal processing (edge-preserving smoothing)assuring a high level of image structure preservation while effectivelyaddressing the issue of residual noise in a regular image such as adigital picture, is realized through the embodiment.

The difference between the first embodiment and the second embodiment isbriefly explained. It has been confirmed through testing that byadjusting the parameter settings, substantially equal levels of noiseremoval effect and countermeasures against residual noise can beachieved through the sequential analysis method and the sequentialsynthesis method. A noteworthy difference between them may be thedifferent orders with which the individual processing phases areexecuted. Namely, the “sequential analysis” method, in which thelow-resolution side is executed at a later stage, is more effective inpreventing leakage of the noise of a long cycle component extracted onthe low resolution side that is bound to affect the processing executedat another resolution. The “sequential synthesis” method, in which thehigh-resolution side is processed at a later stage, is more effective inpreventing leakage of the noise extracted on the high resolution sideand is thus more effective when adopted to extract persistent noise witha Nyquist frequency, such as a checkered pattern.

It is to be noted that in the embodiment, a specific type of noiseremoval processing is executed on the image signal S0(LL0) in the realspace (see FIG. 10). However, a load of the processing executed on theimage signal S0(LL0) in the real space is bound to be extremely heavy.Even through the processing executed at lower resolutions equal to orlower than that corresponding to the image signals S1(LL1, LH1. HL1,HH1) in the embodiment alone, a sufficient level of fine noise removalprocessing is achieved. Accordingly, the specific noise removalprocessing executed on the image signal S0(LL0) in the real space may beskipped. FIG. 17 presents a flowchart of the processing executed on theluminance component (luminance signal) by skipping the specific noiseremoval processing on the image signal S0(LL0) in the real space in FIG.10. The chrominance components, too, may be processed in a similarmanner. Through these measures, fine noise removal processing can beexecuted with a lighter processing load.

THIRD EMBODIMENT

Examples of noise removal processing have been described in reference tothe first and second embodiments. In the third embodiment, the presentinvention is adopted in edge emphasis processing, instead of in noiseremoval processing, so as to facilitate adjustment of the frequencycharacteristics at multiple resolutions.

Since the image processing apparatus in the third embodiment assumes astructure similar to that of the image processing apparatus in the firstembodiment, its explanation is omitted. FIG. 12 presents a flowchart ofthe edge emphasis processing executed in conjunction with multipleresolution transformation. The processing differs from that in thepreceding embodiments only in that it does not include the sequentialprocessing feedback routine executed in the noise removal and that edgecomponent extraction processing, instead of the noise componentextraction processing, is executed. The edge component extractionprocessing may be executed through, for instance, unsharp maskprocessing or band pass filter processing executed on the individualsubband planes.

The edge component extraction processing may be executed by using themultiple-resolution images obtained through transformation for noisecomponent extraction as has been explained in reference to the firstembodiment and the second embodiments, concurrently while the noisecomponent extraction processing is executed, or the edge componentextraction processing may be executed on an image having undergone thenoise removal processing in the first embodiment or the secondembodiment. As an alternative, the edge component extraction processingalone may be executed as long as the target image simply requires edgeemphasis. It is to be noted that the edge component extractionprocessing should, in principle, be executed on the luminance planealone.

While the explanation of the embodiment provided below is simplified byassuming that the edge emphasis processing is executed by itself, it ismore desirable from the viewpoint of image quality, to execute the noiseremoval and the edge emphasis at the same time. More specifically, it isdesirable to synthesize edge components extracted from subband planeshaving undergone the virtual intense noise removal to achieve anoise-free state, as has been explained in reference to the first andsecond embodiments, so as to ensure that the extracted edge componentsdo not contain any noise component and then to incorporate throughaddition processing the synthesized edge component to the image havingundergone the actual noise removal. Accordingly, if the edge emphasis isexecuted in conjunction with the processing in, for instance, the secondembodiment, the synthesis processing on the right hand side in FIG. 10will include three processing phases, 1) noise component synthesis forvirtual noise removal, 2) noise component synthesis for actual noiseremoval and 3) edge component synthesis for actual edge emphasis.

While the present invention, adopted in noise removal processingexecuted in conjunction with multiple resolution transformation,facilitates adjustment of the noise component frequency characteristicsand noise component removal intensity so that the user is able to easilycheck how the target image is visually altered through the noiseremoval. It also provides a system that allows the edge componentfrequency characteristics and intensity to be freely adjusted so thatthe user is able to easily check how the target image is visuallyaffected through the edge emphasis when it is adopted in edge emphasisprocessing executed in conjunction with multiple resolutiontransformation.

The frequency characteristics of the extracted edge components arealtered by adjusting the weights applied to the low-frequency subband(LL) and the high-frequency subbands (LH, HL, HH). FIG. 13 shows theweighting coefficients applied to the low-frequency subband (LL) and thehigh-frequency subbands (LH, HL, HH). k1 may assume different values incorrespondence to LH and HL. The term “low-frequency subband” is used inthis context to refer to a low-frequency edge component image, whereasthe term “high-frequency subbands” is used in this context to refer to ahigh-frequency edge component image.

The low-frequency edge component image and the high-frequency edgecomponent images with the weights applied thereto modulated incorrespondence to the different edge component frequency bands areutilized in inverse wavelet transformation. As shown in FIG. 12, theinverse wavelet transformation (synthesis) is executed repeatedly insequence by using the low-frequency edge component image and thehigh-frequency edge component images with the weights applied theretomodulated at each resolution until a single edge component image, theresolution of which matches that of the original image, is obtained.Finally, based upon the synthesized edge component, the edges in theoriginal image are emphasized.

As in the first embodiment and the second embodiment, edge componentsare extracted from both the high-frequency subbands and thelow-frequency subband at each specific level of resolution among themultiple-resolutions and the edge components having been extracted aresynthesized by applying specific weighting coefficients to the differentsubbands. As a result, the entire edge component frequency band iscovered and an environment in which the frequency characteristics can beadjusted with ease and the visual effect of the edge emphasis can beadjusted is provided.

It is to be noted that in the embodiment, a specific type of edgecomponent extraction processing is executed on the image signal S0(LL0)in the real space (see FIG. 10). However, a load of the processingexecuted on the image signal S0(LL0) in the real space is bound to beextremely heavy. Even through the processing executed at lowerresolutions equal to or lower than that corresponding to the imagesignals S1(LL1, LH1. HL1, HH1) in the embodiment alone, a sufficientlevel of fine edge emphasis processing is achieved. Accordingly, thespecific edge component extraction processing executed on the imagesignal S0(LL0) in the real space may be skipped. FIG. 18 presents aflowchart of the edge emphasis processing executed in conjunction withmultiple resolution transformation by skipping the specific edgecomponent extraction processing on the image signal S0(LL0) in the realspace in FIG. 12. Through these measures, effective emphasis processingcan be executed with a lighter processing load.

(Variations)

It is to be noted that the first through third embodiments have beendescribed by assuming that the multiple resolution transformation isexecuted through wavelet transformation. Instead of wavelettransformation, Laplacian pyramids may be utilized in the multipleresolution transformation. Individual Gaussian pyramids generated duringthe process of Laplacian pyramid generation correspond to thelow-frequency subband (LL) generated through the wavelet transformation,whereas individual Laplacian pyramids correspond to the high-frequencysubbands (LH, HL, HH) generated through the wavelet transformation. Itis to be noted that while a low-frequency subband and the correspondinghigh-frequency subbands generated through the wavelet transformationassume matching resolutions, the resolution of the Laplacian bands,i.e., the high-frequency subbands, corresponding to a specific Gaussianband, i.e., the low-frequency subband, is higher by one step relative tothe resolution of the Gaussian band.

Reference “P. H. Burt and E. H. Adelson, “The Laplacian Pyramid as aCompact Image Code”, IEEE Transactions on Communication, vol. 31, No. 4,pp 532˜540, 1983” may be referred to for further details on Laplacianpyramids.

In addition, instead of the Laplacian pyramid representation, aSteerable pyramid (Steerable wavelet transformation, directional wavelettransformation) representation may be adopted in the multiple resolutiontransformation. In the Steerable pyramid representation, too, theGaussian band generated through Laplacian pyramid generation directlycorresponds to the low-frequency subband. However, while a single typeof isotropic high pass component is generated as a Laplacian bandcorresponding to a high-frequency subband in the Laplacian pyramidrepresentation, a plurality of Laplacian bands with anisotropic highpass components taken along a plurality of directions correspond tohigh-frequency subbands in the Steerable pyramid representation.

Reference “W. T. Freeman and E. H. Adelson, “The Design and Use ofSteerable Filters”, IEEE Transaction On Pattern and MachineIntelligence, Vol 13 No. 9, pp. 891˜906, September 1991” may be referredto for further details on Steerable filters.

FIG. 14 presents a schematic diagram indicating the correspondencebetween the low-frequency subband and the high-frequency subbands ineach type of multiple-resolution representations among the orthogonalwavelet transformation, the Laplacian pyramid and the Steerable pyramid.

In the first embodiment, noise is removed from both the luminancecomponent and the chrominance components through the “sequentialanalysis” method, whereas in the second embodiment, noise is removedfrom both the luminance component and the chrominance components throughthe “sequential synthesis” method. Instead, noise may be removed fromthe luminance component through the sequential analysis and from thechrominance components through the sequential synthesis. As analternative, noise may be removed from the luminance component throughthe sequential synthesis and from the chrominance components through thesequential analysis.

In the first embodiment described earlier, the band-limited images ineach band resulting from the wavelet transformation individually undergovirtual noise removable processing, noise components are extracted fromthe band-limited images having undergone the virtual noise removableprocessing, the extracted noise components are synthesized throughinverse wavelet transformation and the actual noise-free image isgenerated by subtracting the synthesized noise components from the imagein the real space. As an alternative, virtual noise removal processingmay be executed based upon the individual band-limited images in eachband, noise components may be extracted from the band-limited imageshaving undergone the virtual noise removal processing, actual noise-freeimages may be generated by using the extracted noise components, thenoise-free images thus generated may be synthesized through inversewavelet transformation and the synthesized noise-free image may beoutput as the actual noise-free image.

FIG. 20 presents a flowchart of such noise removal processing executedon the luminance component (luminance signal). FIG. 20 corresponds toFIG. 3 in reference to which the first embodiment has been described.The processing executed at, for instance, the ½ resolution is identicalto that executed in the first embodiment up to the point at which thenoise components are extracted in processing (1-5). Subsequently, inprocessing (1-8), actual noise-free image signals S1″(LL1″, LH1″, HL1″,HH1″) are generated by subtracting noise components n1′ obtained byapplying specific weights k0:1:1:1 to the extracted noise components n1from the image signals S1(LL1, LH1, HL1, HH1). During this process,noise removal is executed after applying a weighting coefficientparameter λ, i.e., the noise removal factor, to the extracted noisecomponents n1′ so as to enable adjustment of the extent of noise removalfor the entire image. The weighting coefficient parameter λ is similarto the weighting coefficient parameter λ used in the final noise removalprocessing executed in the real space in the first embodiment.

In addition, since S2′″(LL1) obtained by synthesizing the noisecomponents from the subbands at the lower resolution is multiplied bythe coefficient β(2), LL1″ is multiplied by a coefficient (1−β(2)) inprocessing (1-9). Then, in processing (1-10) the following arithmeticoperation is executed.

S1″((1−β(2))×LL1″,LH1″,HL1″,HH1″)+S2′″(β(2)×LL1)

Subsequently, inverse wavelet transformation is executed on the resultsof processing (1-10), so as to generate a signal with the resolutionhigher by one stage, i.e., the signal S1′″(LL0) with the resolution inthe real space in this example. This processing is sequentially executedstarting at the lowermost layer resolution and, ultimately, a noise-freeimage S0 NR obtained by synthesizing all the band-limited images havingundergone the actual noise removal, including the band-limited images atthe real space resolution, is output.

Similar results can be achieved either by synthesizing the noisecomponents having been extracted in the individual bands resulting fromthe wavelet transformation, subtracting the synthesized noise componentfrom the image in the real space so as to generate the actual noise-freeimage, as has been explained earlier, or by generating actual noise-freeimages in the individual bands and synthesizing the noise-free imagesthus generated through inverse wavelet transformation to generate anactual noise-free image, as explained in reference to the currentvariation.

It is to be noted that examples of possible variations of the firstembodiments have been described so far, the second embodiments and thethird embodiments allow for similar variations.

While the processing is executed by the personal computer 1 in theembodiments described above, the present invention is not limited tothis example. For instance, the present invention may be adopted inprocessing executed in an image-capturing device such as a camera, or itmay be adopted in another type apparatus. In other words, the presentinvention can be adopted in all types of apparatus that handle imagedata.

While two different types of noise removal, i.e., the virtual noiseremoval and the actual noise removal, are executed during the process ofsynthesizing the noise components provided through two systems in thesequential synthesis explained earlier in reference to the embodiments,the present invention is not limited to this example. For instance, whenreconstructing an image by synthesizing noise-free subbands images, twodifferent types of noise-free subbands images may be generated to beindividually synthesized through two separate systems as disclosed inpatent references 5 (U.S. Pat. No. 5,526,446) and 9 (U.S. Pat. No.6,937,772) in the related art.

While the modified bilateral filter and the Laplacian noise extractionmethod have been described earlier as typical examples of noise removalprocessing, the noise removal processing may be executed by usinganother type of noise removal filter.

Although various embodiments and variations have been explained in theabove description, the present invention is not to be considered to belimited to the details thereof. Other possibilities that may beconsidered to fall within the range of the technical concept of thepresent invention are also included within the scope of the presentinvention.

The disclosure of the following priority application is hereinincorporated by reference:

Japanese Patent Application No. 2006-096986 filed Mar. 31, 2006

1. An image processing method adopted to remove noise present in animage, comprising: an image input step in which an original imageconstituted of a plurality of pixels is input; a multiple-resolutionimages generation step in which a plurality of band-limited images withresolutions decreasing in sequence are generated by filtering the inputoriginal image; a first noise removal step in which virtual noiseremoval processing is executed individually for each of the band-limitedimages; a second noise removal step in which actual noise removalprocessing is executed for the individual band-limited images based uponthe band-limited images from which noise has been virtually removedthrough the first noise removal step; and an image acquisition step inwhich a noise-free image of the original image is obtained based uponthe individual band-limited images from which noise has been actuallyremoved through the second noise removal step, wherein: the virtualnoise removal processing executed in the first noise removal step andthe actual noise removal processing executed in the second noise removalstep are differentiated in correspondence to a frequency band of aband-limited image.
 2. An image processing method according to claim 1,wherein: noise is removed to a greater extent in the virtual noiseremoval processing executed in the first noise removal step relative tothe actual noise removal processing executed in the second noise removalstep.
 3. An image processing method adopted to remove noise present inan image, comprising: an image input step in which an original imageconstituted of a plurality of pixels is input; a multiple-resolutionimages generation step in which a plurality of band-limited images withresolutions decreasing in sequence are generated by filtering the inputoriginal image; a first noise extraction step in which noise componentscontained in the band-limited images are individually extracted; asecond noise extraction step in which noise components to be reflectedin the original image are re-extracted from the individual band-limitedimages based upon the noise components having been extracted from theband-limited images in the first noise extraction step; a noisesynthesis step in which the noise components having been re-extractedfrom the band-limited images in the second noise extraction step aresynthesized; and an image acquisition step in which a noise-free imageof the original image is acquired based upon the synthesized noisecomponent.
 4. An image processing method according to claim 3, wherein:the noise components are re-extracted intrinsically from the individualband-limited images in correspondence to frequency bands of theband-limited images in the second noise extraction step.
 5. An imageprocessing method according to claim 4, wherein: low-frequency imagesand high-frequency images with resolutions decreasing in sequence aregenerated in the multiple-resolution images generation step; and thenoise components to be reflected in the original image are re-extractedby weighting the noise components having been extracted in the firstnoise extraction step differently between the low-frequency images andthe high-frequency images in the second noise extraction step.
 6. Animage processing method according to claim 4, wherein: the noisecomponents to be reflected in the original image are re-extracted byweighting the noise components having been extracted in the first noiseextraction step differently between band-limited images with varyinglevels of resolution in the second noise extraction step.
 7. An imageprocessing method adopted to remove noise present in an image,comprising: an image input step in which an original image constitutedof a plurality of pixels is input; a multiple-resolution imagesgeneration step in which a plurality of band-limited images withresolutions decreasing in sequence are generated by filtering the inputoriginal image; a noise extraction step in which a noise componentcontained in one of the band-limited images is extracted and then anoise component in another band-limited image is extracted in sequencebased upon the noise component having been extracted; and a noiseremoval step in which noise in the original image is removed based uponnoise components in the individual band-limited images having beenextracted, wherein: a signal intensity level with which the noisecomponent extracted from one band-limited image in the noise extractionstep is reflected when extracting the noise component from anotherband-limited image during the noise extraction step is set differentlyfrom a signal intensity level with which the noise component extractedfrom one band-limited image in the noise extraction step is reflected inthe original image during the noise removal step.
 8. An image processingmethod according to claim 7, wherein: the noise component contained inthe one band-limited image is extracted and a noise component containedin another band-limited image assuming a different resolution isextracted in sequence based upon the noise component having beenextracted in the noise extraction step.
 9. An image processing methodaccording to claim 7, wherein: low-frequency images and high-frequencyimages with resolutions decreasing in sequence are generated in themultiple-resolution images generation step.
 10. An image processingmethod adopted to remove noise present in an image, comprising: an imageinput step in which an original image constituted of a plurality ofpixels is input; a multiple-resolution images generation step in whichsets of band-limited images each constituted with at least two types ofband-limited images are generated in sequence over a plurality ofresolutions by filtering the input original image, generating a set ofband-limited images constituted with at least two different types ofband-limited images and then repeatedly filtering at least one type ofband-limited images among the band-limited images; a noise extractionstep in which processing whereby noise components contained in the twotypes of band-limited images are extracted and a noise component in atleast one type of band-limited image at another resolution is extractedbased upon the extracted noise components, is executed in sequence so asto extract noise components in the individual band-limited images atvarious resolutions; and a noise removal step in which noise in theoriginal image is removed from based upon noise components in theindividual band-limited images having been extracted, wherein: aninter-band signal intensity with which a set of noise components of atleast two types of band-limited images obtained through the noiseextraction step is reflected when extracting the noise component inanother band-limited image in the noise extraction step is setdifferently from an inter-band signal intensity with which the set ofnoise components of at least two types of band-limited images obtainedthrough the noise extraction step is reflected in the original imageduring the noise removal step.
 11. An image processing method accordingto claim 7, wherein: two different types of images that arelow-frequency images and high-frequency images are generated to make upthe sets of band-limited images constituted with at least two differenttypes of band-limited images in the multiple resolution image generationstep.
 12. An image processing method adopted to remove noise present inan image, comprising: an image input step in which an original imageconstituted of a plurality of pixels is input; a multiple-resolutionimages generation step in which one or more low-frequency images withresolutions decreasing in sequence and one or more high-frequency imageswith the resolutions decreasing in sequence are generated by decomposingthe input original image; a noise modulation step in which a noisecomponent contained in each low-frequency image and a noise componentcontained in each high-frequency image are individually extracted, alow-frequency noise image and a high-frequency noise image are generatedin correspondence to the low-frequency image and the high-frequencyimage and weights applied to the noise components over differentfrequency bands are modulated by applying a weighting coefficient to atleast either the low-frequency noise image or the high-frequency noiseimage having been generated; a noise synthesis step in which thelow-frequency noise image and the high-frequency noise image havingundergone the noise modulation step are combined to generate a singlesynthesized noise image with higher resolution by one stage and noiseimages are repeatedly synthesized in sequence until a single noise imagesignal with a resolution matching the resolution of the original imageis obtained; and a noise removal step in which noise contained in theoriginal image is removed based upon the synthesized noise image signal.13. An image processing method adopted to remove noise present in animage, comprising: an image input step in which an original imageconstituted of a plurality of pixels is input; a multiple-resolutionimages generation step in which one or more low-frequency images havingresolutions decreasing in sequence and one or more high-frequency imageswith the resolutions decreasing in sequence are generated by decomposingthe input original image; a noise extraction step in which a noisecomponent contained in each low-frequency image and a noise componentcontained in each high-frequency image are individually extracted and alow-frequency noise image and a high-frequency noise image are generatedin correspondence to the low-frequency image and the high-frequencyimage; a noise modulation step in which weights applied to the noisecomponents over different frequency bands are modulated by applying aweighting coefficient to at least either the low-frequency noise imageor the high-frequency noise image having been generated; a noisesynthesis step in which the low-frequency noise image and thehigh-frequency noise image having undergone the noise modulation stepare combined to generate a single synthesized noise image with higherresolution by one stage and the synthesized noise image with the higherresolution is combined with the low-frequency noise image correspondingto the low-frequency image with the higher resolution by one stage so asto generate a new synthesized low-frequency noise image; a noisesynthesis repeating step in which synthesis is repeatedly executed byrepeating a sequence of the noise extraction step, the noise modulationstep and the noise synthesis step until a single noise image signal witha resolution matching the resolution of the original image is obtained;and a noise removal step in which noise contained in the original imageis removed based upon the synthesized noise image signal.
 14. An imageprocessing method adopted to remove noise present in an image,comprising: an image input step in which an original image constitutedof a plurality of pixels is input; a multiple-resolution imagesgeneration step in which one or more low-frequency images withresolutions decreasing in sequence and one or more high-frequency imageswith the resolutions decreasing in sequence are generated by decomposingthe input original image; a noise extraction step in which a noisecomponent contained in each low-frequency image and a noise componentcontained in each high-frequency image are individually extracted and alow-frequency noise image and a high-frequency noise image are generatedin correspondence to the low-frequency image and the high-frequencyimage; a noise modulation step in which weights applied to the noisecomponents over different frequency bands are modulated by applying aweighting coefficient to at least either the low-frequency noise imageor the high-frequency noise image having been generated; a noisesynthesis step in which the low-frequency noise image and thehigh-frequency noise image having undergone the noise modulation stepare combined to generate a single synthesized noise image with higherresolution by one stage; and a noise removal step in which noisecontained in the original image is removed based upon the synthesizednoise image.
 15. An image processing method adopted to remove noisepresent in an image, comprising: an image input step in which anoriginal image constituted of a plurality of pixels is input; amultiple-resolution images generation step in which one or morelow-frequency images with resolutions decreasing in sequence and one ormore high-frequency images with the resolutions decreasing in sequenceare generated by decomposing the input original image; a noiseextraction step in which a noise component contained in eachlow-frequency image and a noise component contained in eachhigh-frequency image are individually extracted and a low-frequencynoise image and a high-frequency noise image are generated incorrespondence to the low-frequency image and the high-frequency image;a noise modulation step in which weights applied to the noise componentsover different frequency bands are modulated by applying a weightingcoefficient to at least either the low-frequency noise image or thehigh-frequency noise image having been generated; a noise synthesis stepin which the low-frequency noise image and the high-frequency noiseimage having undergone the noise modulation step are combined togenerate a single synthesized noise image with higher resolution by onestage and the synthesized noise image with the higher resolution iscombined with the low-frequency noise image corresponding to thelow-frequency image with the higher resolution by one stage so as togenerate a new synthesized low-frequency noise image; and a noiseremoval step in which noise contained in the original image is removedbased upon the synthesized noise image signal.
 16. An image processingmethod according to claim 12, wherein: the low-frequency noise image andthe high-frequency noise image corresponding to the low-frequency imageand the high-frequency image are generated based upon observation oflocal signal values in the low-frequency image and the high-frequencyimage in the noise extraction step.
 17. An image processing methodadopted to emphasize edges in an image, comprising: an image input stepin which an original image constituted of a plurality of pixels isinput; a multiple-resolution images generation step in which one or morelow-frequency images with resolutions decreasing in sequence and one ormore high-frequency images with the resolutions decreasing in sequenceare generated by decomposing the input original image; an edge componentgeneration step in which an edge component is extracted by applying aband pass filter to each low-frequency image and an edge component isextracted by applying a band pass filter to each high-frequency imageand a low-frequency edge component image and a high-frequency edgecomponent image are generated in correspondence to the low-frequencyimage and the high-frequency image; an edge component modulation step inwhich weights applied to the edge components over different frequencybands are modulated by applying a weighting coefficient to at leasteither the low-frequency edge component image or the high-frequency edgecomponent image having been generated; an edge component synthesis stepin which the low-frequency edge component image and the high-frequencyedge component image having undergone the edge component modulation stepare combined to generate a single synthesized edge component image withhigher resolution by one stage and edge component images are repeatedlysynthesized in sequence until a single edge component image with aresolution thereof matching the resolution of the original image isobtained; and an edge emphasis step in which edges contained in theoriginal image are emphasized based upon the synthesized edge componentimage.
 18. An image processing method adopted to emphasize edges in animage, comprising: an image input step in which an original imageconstituted of a plurality of pixels is input; a multiple-resolutionimages generation step in which one or more low-frequency images havingresolutions decreasing in sequence and one or more high-frequency imageswith the resolutions decreasing in sequence are generated by decomposingthe input original image; an edge component generation step in which anedge component is extracted by applying a band pass filter to eachlow-frequency image and an edge component is extracted by applying aband pass filter to each high-frequency image and a low-frequency edgecomponent image and a high-frequency edge component image are generatedin correspondence to the low-frequency image and the high-frequencyimage; an edge component modulation step in which weights applied to theedge components over different frequency bands are modulated by applyinga weighting coefficient to at least either the low-frequency edgecomponent image or the high-frequency edge component image having beengenerated; an edge component synthesis step in which the low-frequencyedge component image and the high-frequency edge component image havingundergone the edge component modulation step are combined to generate asingle synthesized edge component image with a higher resolution by onestage; and an edge emphasis step in which edges contained in theoriginal image are emphasized based upon the synthesized edge componentimage.
 19. An image processing method according to claim 5, wherein: thelow-frequency image and the high-frequency image correspond to; 1) alow-frequency component and a high-frequency component generated throughorthogonal wavelet transformation; 2) a Gaussian component and aLaplacian component in a Laplacian pyramid representation; or 3) alow-frequency component and high-frequency components each correspondingto a specific direction in directional wavelet transformation.
 20. Animage processing method according to claim 19, wherein: whenmultiple-resolution images are generated through two-dimensionalorthogonal wavelet transformation, the low-frequency image correspondsto an LL subband and the high-frequency image corresponds to an LHsubband, an HL subband or an HH subband.
 21. An image processing methodadopted to remove noise present in an image, comprising: inputting anoriginal image constituted of a plurality of pixels; sequentiallygenerating images with varying resolutions by that the input originalimage undergoes multiple resolution transformation; extracting a noisecomponent by using an image generated at a given resolution; using thenoise component extracted at the given resolution for purposes ofextracting a noise component at another resolution and synthesizing anoise component to be removed from the original image through inversemultiple resolution transformation; and applying different weights tothe noise component extracted at the given resolution to be used forpurposes of extracting the noise component at the other resolution andto the noise component extracted at the given resolution to be used tosynthesize the noise component to be removed from the original imagethrough inverse multiple resolution transformation.
 22. An imageprocessing program enabling a computer or an image processing apparatusto execute an image processing method according to claim
 1. 23. An imageprocessing apparatus, comprising: an image processing program accordingto claim 22 installed therein.